Why is my multibranch project cloning the whole repository on the controller?

Article ID:115002991272
3 minute readKnowledge base

Issue

  • My multibranch pipeline clones the entire repository on the controller to obtain the Jenkinsfile

  • My multibranch pipeline shows a timeout error when cloning the repository to obtain the Jenkinsfile:

Branch indexing
23:43:32 Connecting to https://api.github.com using my-credentials/****** (my-credentials GitHub personal access token)
Checking out git https://github.com/org/repo.git into /var/jenkins_home/workspace/_repo_PR-123-P7DALB4WUZUT3GSAR3HPW2RCNJME43BF5FNLWOCKUK55BAHQV6SA@script to read Jenkinsfile
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository https://github.com/org/repo.git
 > git init /var/jenkins_home/workspace/_repo_PR-123-P7DALB4WUZUT3GSAR3HPW2RCNJME43BF5FNLWOCKUK55BAHQV6SA@script # timeout=10
Fetching upstream changes from https://github.com/org/repo.git
 > git --version # timeout=10
using GIT_ASKPASS to set credentials my-credentials GitHub personal access token
 > git fetch --no-tags --progress https://github.com/org/repo.git +refs/pull/123/head:refs/remotes/origin/PR-123 +refs/heads/master:refs/remotes/origin/master
ERROR: Timeout after 10 minutes
[...]

Environment

Related Issues

  • JENKINS-33273: Optimize Jenkinsfile loading and branch detection

  • JENKINS-43194: Lightweight checkout for PR merge jobs

  • JENKINS-42518: Provide lightweight checkout capability for bitbucket to avoid repository clone for multi-branch pipeline jobs

Resolution

Pipeline from SCM used to perform several checkout as explained in JENKINS-33273. Since version 2.13 of Pipeline Multibranch Plugin and the resolution of JENKINS-33273, Pipeline from SCM offers the ability to perform a lightweight checkout - if the SCM supports it - to retrieve the Jenkinsfile. With a lightweight checkout, no clone is needed to retrieve the Jenkinsfile anymore. The Jenkinsfile is instead retrieved using an API.

The following plugin provide support for lightweight checkout:

However in the case of merge Pull Requests only those versions of the plugins support a lightweight checkout:

In unsupported versions, plugin falls back to the previous behavior - a heavyweight checkout - which mean that a full clone is performed to retrieve the Jenkinsfile.

For such cases, there are workarounds that can be used until further support for a lightweight checkout is provided.

Workaround 1: Use a reference Repository

This workaround requires SCM API v2.2.0: Prior to that version, only the behavior of the checkout step can be adjusted - see [How to Customize Checkout for Pipeline Multibranch? - but not the behavior of the checkout carried out to obtain the Jenkinsfile.

A reference repository is a general tip to speed up the checkout/cloning of large Git repository. You can set up a reference repository on the controller that will serve as a cache for your repository. It also saves disk space and reduce the traffic between Jenkins and the Git server.

First, have a look at the article Using a Git reference repository to understand how to set up a reference repository.

Then, in the configuration of the organization / multibranch project, add an SCM Trait of type Advanced Clone Behaviors and specify the path to the reference repository:

scm trait git ref repo config

Workaround 2: Increase the Git timeout

If affected by the timeout error, it is possible to increase the Git timeout.

You can increase the timeout globally by setting the system property -Dorg.jenkinsci.plugins.gitclient.Git.timeOut=<number of minutes> on startup. Upon restart, you can check that the timeout is properly set by running the groovy script return org.jenkinsci.plugins.gitclient.CliGitAPIImpl.TIMEOUT under Manage Jenkins  Script Console.

SCM API v2.2.0 and later

You can set the timeout per organization / multibranch. In the configuration of organization / multibranch project, add the SCM Traits of type Advanced Clone Behaviors and Advanced Checkout Behaviors and set a suitable value for the timeout (in minutes):

scm trait git timeouts

About SCM API v2.2.0

Since version 2.2.0 of the SCM API Plugin, it is now possible to add SCM Traits - like the git additional behaviors - at the organization / multibranch level. These traits apply to:

  • the checkout performed to retrieve the Jenkinsfile

  • the checkout scm step

SCM API v2.2+ along with GitHub Branch Source 2.2+ and Bitbucket Branch Source 2.2+ are available under the CloudBees Assurance Program since CJE 2.73.3.1.

About Multibranch Pipeline support

Multibranch Pipelines support lightweight checkout since JENKINS-33273. It is activated by default. There is a kill switch to deactivate it. Simply add org.jenkinsci.plugins.workflow.multibranch.USE_HEAVYWEIGHT_CHECKOUT=true to Jenkins arguments (please refer to How to add Java arguments to Jenkins?).

Why is my Pipeline job cloning the entire repository on the controller even when I select a lightweight checkout?

In order to work the lightweight checkout needs a cache repository. This cache repository is created the first time on the controller.

Why is my Multibranch job cloning the entire repository on the controller ?

Like for a standard Pipeline job, lightweight checkout needs a cache repository. This is used for instance by the branch indexing feature.

This article is part of our Knowledge Base and is provided for guidance-based purposes only. The solutions or workarounds described here are not officially supported by CloudBees and may not be applicable in all environments. Use at your own discretion, and test changes in a safe environment before applying them to production systems.