Generating a support bundle

The CloudBees Support plugin provides the ability to generate a bundle of all the commonly-requested diagnostic information used by CloudBees when resolving support issues. It extends the open source Support Core plugin, which is installed as a dependency when the CloudBees Support plugin is installed.

The CloudBees Support plugin is installed by default with CloudBees Core, CloudBees Jenkins Distribution, and CloudBees Jenkins Platform. If you are a CloudBees Jenkins Support customer using the Jenkins LTS please install the Jenkins Support Core plugin instead.

This guide explains what a support bundle is and how to generate, both from the UI and the CLI.

What is a support bundle

A bundle is a simple .zip file containing mostly plain text files from the root of your Jenkins home directory, $JENKINS_HOME, which can be used to assist a CloudBees Support Engineer in diagnosing your issue. You can inspect the contents of this file to assure yourself that it does not contain information you do not want to share with CloudBees. You can even unpack and repack the bundle if there is some specific piece of information that you need to remove from the bundle.

What a support bundle included

In summary, the contents of a support bundle contains:

The following are not included in a support bundle:

  • Credentials

  • Usernames (no, except jenkins user)

  • Job Details

  • Groovy Pipeline Code

An example filename of a support bundle is cloudbees-support_core-cm-master.example.com_2019-09-23_19.55.52.zip.

The following is an example of the zip file contents:

├── about.md (<-- Hostname)
├── admin-monitors.md
├── browser.md
├── buildqueue.md
├── cap
│   ├── beekeeper.md
│   └── properties.md
├── docker
│   └── Dockerfile
├── items.md
├── load-stats
│   ├── label
│   │   └── master
│   │       ├── gnuplot
│   │       ├── hour.csv
│   │       ├── min.csv
│   │       └── sec10.csv
│   ├── no-label
│   │   ├── gnuplot
│   │   ├── hour.csv
│   │   ├── min.csv
│   │   └── sec10.csv
│   └── overall
│       ├── gnuplot
│       ├── hour.csv
│       ├── min.csv
│       └── sec10.csv
├── loggers.md
├── manifest.md
├── node-monitors.md
├── nodes
│   └── master
│       ├── checksums.md5
│       ├── dmesg.txt
│       ├── dmi.txt
│       ├── environment.txt
│       ├── file-descriptors.txt
│       ├── logs (<-- Hostname, IP Addresses)
│       │   ├── all_2019-09-23_19.29.56.log
│       │   ├── all_2019-09-23_19.33.20.log
│       │   ├── all_2019-09-23_19.50.20.log
│       │   ├── all_memory_buffer.log
│       │   └── jenkins.log
│       ├── metrics.json
│       ├── networkInterface.md (<-- IP Address)
│       ├── oome.md
│       ├── pipeline-thread-dump.txt
│       ├── pipeline-timings.txt
│       ├── proc
│       │   ├── cpuinfo.txt
│       │   ├── meminfo.txt
│       │   ├── mounts.txt
│       │   ├── net
│       │   │   └── rpc
│       │   │       ├── nfs.txt
│       │   │       └── nfsd.txt
│       │   ├── self
│       │   │   ├── cmdline (<-- Hostname)
│       │   │   ├── environ (<-- Hostname, Username)
│       │   │   ├── limits.txt
│       │   │   ├── mountstats.txt
│       │   │   └── status.txt
│       │   ├── swaps.txt
│       │   └── system-uptime.txt
│       ├── sysctl.txt
│       ├── system.properties
│       ├── thread-dump.txt
│       └── userid.txt (<-- Username)
├── nodes.md
├── other-logs
│   ├── Argus\ Notifier\ Periodic\ Sender.log
│   ├── Argus\ Notifier\ Periodic\ Sender.log.1
│   ├── Download\ metadata.log
│   ├── Download\ metadata.log.1
│   ├── Download\ metadata.log.2
│   ├── SharedConfigurationSynchronizer.log
│   ├── SharedConfigurationSynchronizer.log.1
│   ├── SharedConfigurationSynchronizer.log.2
│   ├── Template\ Catalogs\ Update.log
│   ├── Template\ Catalogs\ Update.log.1
│   ├── com.cloudbees.jenkins.plugin.metrics.views.Alerter.log
│   ├── com.cloudbees.jenkins.plugin.metrics.views.Alerter.log.1
│   ├── com.cloudbees.jenkins.support.impl.cloudbees.IOPerfPeriodicWork.log
│   ├── com.cloudbees.jenkins.support.impl.cloudbees.IOPerfPeriodicWork.log.1
│   ├── com.cloudbees.opscenter.client.cloud.CloudImpl$PeriodicWorkImpl.log
│   ├── com.cloudbees.opscenter.client.cloud.CloudImpl$PeriodicWorkImpl.log.1
│   ├── copy_reference_file.log
│   └── health-checker.log
├── plugins
│   ├── active.txt
│   ├── disabled.txt
│   └── failed.txt
├── update-center.md
└── user.md

16 directories, 78 files

Best practices when generating a support bundle

To provide you a better support service, please consider the following recommendations:

  • Review steps described in Prepare Jenkins for Support.

  • “About Jenkins” is mandatory.

  • The closer the Bundle generation is to the issue, the better for CloudBees Support to diagnose it.

  • In case you are sending a series of support bundles, please attach them separately (by default `.zip format). Do not compress them into a unique package.

  • If your support bundle has a big file size, you could “recompress” a zip with xz. See Sending a support bundle too large for Zendesk for details.

  • If you are experiencing an issue regarding an Operations Center - Client Master connectivity problem, send support bundles for each of the instances.

  • To avoid noise on logs, indicate on the ticket the date and time when the particular issue happened/arose.

  • If you can reproduce the issue by yourself, indicate the followed steps before generating the support bundle.

Generating a support bundle from the UI

To generate a bundle, go to the Support link in the web UI.

generate bundle
Figure 1. Generating a bundle

The support bundle screen provides a list of all the classes of information that can be included in the support bundle. Normally it is best to include all the selected information, however there may be some information which you do not want to share for reasons of confidentiality. In such cases you can de-select the information you do not want to include in the bundle.

Click the Generate Bundle to download the bundle to your machine. A bundle is a simple .zip file containing mostly plain text files. You can inspect the contents of this file to assure yourself that it does not contain information you do not want to share with CloudBees. You can even unpack and repack the bundle if there is some specific piece of information that you need to remove from the bundle.

When you are happy with the bundle, attach the bundle to your CloudBees support ticket.

Generating a support bundle from the CLI

If there is a problem accessing theweb UI, of course it is going to be difficult to diagnose those by going to that same UI and getting a bundle. As an alternative, you can use the Jenkins CLI to obtain a bundle from a shell. Before you encounter problems, make sure you have downloaded jenkins-cli.jar from the server (go to /cli/ to get a link). You may also need to make sure you have authenticated to the CLI, for example by uploading an SSH public key to your Jenkins user account.

Then try:

java -jar /path/to/jenkins-cli.jar -s http://server/ support > ./path/to/support-bundle.zip

This command generates the support bundle and saves it to the path you provided.

Alternatively, if you are using the deprecated and insecure legacy remoting-based CLI, the generated support bundle is saved to a file in your default temporary folder using the filename printed.

You can also request only specific components to include; to see the currently available list, ask for the command’s help:

java -jar /path/to/jenkins-cli.jar -s http://server/ help support

Enabling anonymization

The diagnostic information collected can contain sensitive information, but this can be automatically filtered by enabling support bundle anonymization.

If support bundle anonymization is enabled through the global configuration settings, some metadata is replaced with automatically generated anonymized names: nodes, computers, labels, users, items (including jobs), views, hostnames, and IP addresses. If you need to determine the real value for an anonymized one, you can look that up in the support bundle anonymization web page after logging into your product.

When anonymization is disabled, a warning message is shown on the Support web page.

WARNING: Support bundle anonymization is disabled. This can be enabled in the global configuration under Support Bundle Anonymization.

Click the link to Manage Jenkins  Configure System and enable support bundle anonymization.

Anonymize support bundle contents checkbox
If you are using Jenkins Support Core plugin instead of the CloudBees Support plugin, use version 2.53 or higher of the Jenkins Support Core plugin due to the Known Performance issue with Support Bundle Anonymization.

Controlling who can generate support bundles

By default, only Jenkins administrators can generate support bundles. The CloudBees Support Plugin defines an additional security permission within Jenkins (CloudBees Support/DownloadBundle). You can assign this permission to any users that you want to be able to generate support bundles, even the Anonymous user. The content of such bundles can be useful in diagnosing authentication issues that specific users are having.

CloudBees' general recommendation is only to allow trusted users to generate support bundles. Enabling the anonymous user to generate a bundle should be something that is time bounded and used specifically to obtain a bundle from a user who is having issues with authentication.
To reduce the possibility of the support plugin from leaking confidential information about your Jenkins installation, when a non-administrator generates a support bundle, certain specific components are not available for selection. For example, the Environment variables, System properties and Thread dumps components are disabled as these components tend to include information that is often not for general consumption.
generate restricted
Figure 2. Generating a bundle as a user who is not a Jenkins administrator

Viewing Anonymized Mappings

When submitting an anonymized support bundle to your support organization, they may need to ask further details about items with anonymized names. To translate that, navigate to Manage Jenkins  Support Bundle Anonymization.

Support Bundle Anonymization management link

This page contains a table of mappings between original names and their corresponding anonymized versions. This also contains a list of stop words that are ignored when anonymization generates anonymized counterparts. These are common terms in Jenkins that by themselves convey no personal meaning. For example, an agent named "Jenkins" will not be anonymized because "jenkins" is a stop word.

Screenshot of anonymized mappings management page example

Anonymization limitations

Anonymization filters only apply to text files. It cannot handle non-Jenkins URLs, custom proprietary Jenkins plugin names, and exceptions quoting invalid Groovy code in a Jenkins pipeline. The active plugins, disabled plugins, failed plugins, and Dockerfile reports are not anonymized due to several Jenkins plugins and other Java libraries using version numbers that are indistinguishable from IP addresses. These reports are in the files plugins/active.txt, plugins/disabled.txt, plugins/failed.txt, and docker/Dockerfile. These files should all be manually reviewed if you do not wish to disclose the names of custom proprietary plugins.

Generating a support bundle when Jenkins will not start

When the CloudBees Support Plugin is installed, it automatically stores a bundle every hour in $JENKINS_HOME/support. These bundles are purged using an exponential retention strategy so that they do not overflow disk space.

If your Jenkins instance is not starting correctly, this is usually due to some class-loading conflict among the set of plugins that you have installed in your instance. Normally such class-loading conflicts just results in one of your plugins failing to load, however in some extreme cases a plugin failing to load can cause a second plugin to render the UI of your Jenkins instance inaccessible.

In such cases, the latest support bundle (and some representative historical versions) can be very helpful for CloudBees in ensuring a rapid response and restoration of your Jenkins to an accessible state.

Sending a support bundle too large for Zendesk

Files over 20 MB can’t be attached to a support ticket. If this is the case, you have 3 options:

  1. Recompress the file

  2. Use the CloudBees Uploads Service

  3. Split the file

These 3 options are explained in the next sections.

Recompress the file

Using an additional compression with a different algorithm like xz can help you to reduce the size of the archive. For example, on Linux/MacOS after having installed xz with the package manager you are using for your system (aptitude install xz, yum install xz, brew install xz,etc) you can use it like this:

xz -z -9 -e cloudbees-support.zip

CloudBees Uploads Service

Use https://uploads.cloudbees.com/ to upload the file to CloudBees Support.

Split the file

Another way is to split the archive file into separate smaller files and send them to us. The following command split the file mybundle.zip into several files named mybundle.zip.part-01, mybundle.zip.part-02, and so on and whose the size does not exceed 20 MB:

split --numeric-suffixes -b 20m mybundle.zip "mybundle.zip.part-"

To reassemble the split archive and access the files within:

cat mybundle.zip.part-* > mybundle.zip
unzip -p mybundle.zip > mybundle.tar.xz
mkdir mybundle
tar -xf mybundle.tar.xz -C mybundle

Troubleshooting

The following sections are common problems you might encounter when generating a support bundle and how to resolve them.

The support bundle generated is missing 'pipeline-timings.txt'

If the support bundle is missing pipeline-timings.txt, the Jenkins logs shows an exception like the following:

Apr 23, 2018 2:19:06 PM WARNING com.cloudbees.jenkins.support.SupportPlugin writeBundle
Could not attach 'nodes/master/pipeline-timings.txt' to support bundle
java.io.IOException: <jobName> <buildNumber> did not yet start
	at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:1058)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$PipelineTimings$1.writeTo(CpsFlowExecution.java:1779)
	at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:357)
	at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:271)
	at com.cloudbees.jenkins.support.SupportPlugin$PeriodicWorkImpl$1.run(SupportPlugin.java:741)
	at java.lang.Thread.run(Thread.java:748)

This issue happens if pipeline timings are not available for a specific build at the time that the bundle is generated. This is causes by a non-graceful exception handing in the Pipeline Groovy plugin, that has been fixed in version 2.68 of this plugin.

Upgrade the Pipeline Groovy plugin to version 2.68 or later.

The support bundle generated is missing its Manifest

If the support bundle generated misses some files, you might see something similar to:

2019-07-02 17:06:31.129+0000 [id=847267]	WARNING	c.c.j.s.SupportPlugin$PeriodicWorkImpl#lambda$doRun$0: Could not save support bundle
    Command Close created at
        at hudson.remoting.Command.<init>(Command.java:68)
        at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1265)
        at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1263)
        at hudson.remoting.Channel.close(Channel.java:1436)
        at hudson.remoting.Channel.close(Channel.java:1403)
        at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1270)
    Caused: hudson.remoting.Channel$OrderlyShutdown
        at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1271)
        at hudson.remoting.Channel$1.handle(Channel.java:565)
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:85)
    Caused: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on my-node failed. The channel is closing down or has closed down
        at hudson.remoting.Channel.call(Channel.java:948)
        at hudson.FilePath.act(FilePath.java:1072)
        at hudson.FilePath.act(FilePath.java:1061)
        at hudson.FilePath.lastModified(FilePath.java:1599)
        at com.cloudbees.jenkins.support.api.FilePathContent.getTime(FilePathContent.java:85)
        at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:301)
        at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:276)
        at com.cloudbees.jenkins.support.SupportPlugin$PeriodicWorkImpl.lambda$doRun$0(SupportPlugin.java:764)
        at com.cloudbees.jenkins.support.SupportPlugin$PeriodicWorkImpl.dt_access$313(SupportPlugin.java)
        at java.lang.Thread.run(Thread.java:748)

The support bundle generation may be interrupted by remoting issues while retrieving files on remote agents. When this happens, the bundle generation stops downloading more components but create the bundle nonetheless. This results in the support bundle being incomplete, missing a lot files, in particular its manifest.

The problem happens when getting files information on remote agents and could be avoided by disabling the following options in the Support page:

  • Agent JVM process system metrics (Linux only)

  • Agent system configuration (Linux only)

Upgrade the Jenkins Support Core plugin to version 2.58 or later to fix this issue.

Since enabling/turning on the Support bundle Anonymization feature, Jenkins has been running very slowly

This issue was caused by searching over the entire support bundle for sensitive information when there were many files which would never contain the information that needed to be redacted.

To resolve this issue, update to version 2.53 of the Jenkins Support Core plugin.