Generating a support bundle

6 minute readTroubleshooting

The CloudBees Support plugin provides the ability to generate a bundle of all the commonly-requested diagnostic information used by CloudBees when resolving support issues. It extends the open source Support Core plugin, which is installed as a dependency when the CloudBees Support plugin is installed.

The CloudBees Support plugin is installed by default with CloudBees CI.

This guide explains what a support bundle is and how to generate, both from the UI and the CLI.

What is a support bundle

A bundle is a simple .zip file containing mostly plain text files from the root of your Jenkins home directory, $JENKINS_HOME, which can be used to assist a CloudBees Support Engineer in diagnosing your issue. You can inspect the contents of this file to assure yourself that it does not contain information you do not want to share with CloudBees. You can even unpack and repack the bundle if there is some specific piece of information that you need to remove from the bundle.

What a support bundle includes

In summary, a support bundle contains:

The following are not included in a support bundle:

  • Credentials

  • Usernames (no, except jenkins user)

  • Job Details

  • Groovy Pipeline Code

An example filename of a support bundle is cloudbees-support_core-cm-controller.example.com_2022-11-09_22.55.52.zip.

The following is an example of the zip file contents:

├── about.md (<-- Hostname) ├── admin-monitors.md ├── agent-protocols.md ├── browser.md ├── buildqueue.md ├── cap │ ├── beekeeper.md │ └── properties.md ├── docker │ └── Dockerfile ├── identity.md ├── items.md ├── load-stats │ ├── label │ │ └── label_dark_researcher │ │ ├── gnuplot │ │ ├── hour.csv │ │ ├── min.csv │ │ └── sec10.csv │ ├── no-label │ │ ├── gnuplot │ │ ├── hour.csv │ │ ├── min.csv │ │ └── sec10.csv │ └── overall │ ├── gnuplot │ ├── hour.csv │ ├── min.csv │ └── sec10.csv ├── loggers.md ├── manifest.md ├── node-monitors.md ├── nodes │ └── master │ ├── checksums.md5 │ ├── dmesg.txt │ ├── dmi.txt │ ├── environment.txt │ ├── file-descriptors.txt │ ├── logs (<-- Hostname, IP Addresses) │ │ ├── all_2022-11-09_22.32.49.log │ │ ├── all_memory_buffer.log │ │ └── jenkins.log │ ├── metrics.json │ ├── networkInterface.md (<-- IP Address) │ ├── oome.md │ ├── pipeline-thread-dump.txt │ ├── pipeline-timings.txt │ ├── proc │ │ ├── cpuinfo.txt │ │ ├── meminfo.txt │ │ ├── mounts.txt │ │ ├── net │ │ │ └── rpc │ │ │ ├── nfs.txt │ │ │ └── nfsd.txt │ │ ├── self │ │ │ ├── cmdline (<-- Hostname) │ │ │ ├── environ (<-- Hostname, Username) │ │ │ ├── limits.txt │ │ │ ├── mountstats.txt │ │ │ └── status.txt │ │ ├── swaps.txt │ │ └── system-uptime.txt │ ├── sysctl.txt │ ├── system.properties │ ├── thread-dump.txt │ └── userid.txt (<-- Username) ├── nodes.md ├── operations-center-connector-logs │ ├── connection-details.md │ ├── operations-center-connector.log ├── other-logs │ ├── copy_reference_file.log ├── plugins │ ├── active.txt │ ├── backup.txt │ ├── disabled.txt │ └── failed.txt ├── reverse-proxy.md ├── running-builds.txt ├── task-logs │ ├── com.cloudbees.opscenter.client.cloud.CloudImpl$PeriodicWorkImpl.log │ ├── health-checker.log │ ├── operations-center-connector.log │ ├── SharedConfigurationSynchronizer.log ├── update-center.md ├── user.md └── users └── count.md 16 directories, 78 files

Best practices when generating a support bundle

To provide you a better support service, please consider the following recommendations:

  • Review steps described in Prepare CloudBees CI for Support.

  • “About Jenkins” is mandatory.

  • The closer the Bundle generation is to the issue, the better for CloudBees Support to diagnose it.

  • In case you are sending a series of support bundles, please attach them separately (in .zip format, by default). Do not compress them into a unique package.

  • If your support bundle has a big file size, you could “recompress” a zip with xz. See Sending a support bundle too large for Zendesk for details.

  • If you are experiencing an issue regarding the operations center - client controller connectivity problem, send support bundles for each of the instances.

  • To avoid noise on logs, indicate on the ticket the date and time when the particular issue happened/arose.

  • If you can reproduce the issue by yourself, indicate the followed steps before generating the support bundle.

Generating a support bundle from the UI

To generate a support bundle, navigate to the operations center or controller dashboard, and then select Support.

Figure 1. Generating a bundle
Figure 1. Generating a bundle

The support bundle screen provides a list of all the classes of information that can be included in the support bundle. Normally it is best to include all the selected information, however there may be some information which you do not want to share for reasons of confidentiality. In such cases you can de-select the information you do not want to include in the bundle.

Click the Generate Bundle to download the bundle to your machine. A bundle is a simple .zip file containing mostly plain text files. You can inspect the contents of this file to assure yourself that it does not contain information you do not want to share with CloudBees. You can even unpack and repack the bundle if there is some specific piece of information that you need to remove from the bundle.

When you are happy with the bundle, attach the bundle to your CloudBees support ticket.

Generating a support bundle from the CLI

If there is a problem accessing the web UI, of course it is going to be difficult to diagnose those by going to that same UI and getting a bundle. As an alternative, you can use the Jenkins CLI to obtain a bundle from a shell. Before you encounter problems, make sure you have downloaded jenkins-cli.jar from the server (go to /cli/ to get a link). You may also need to make sure you have authenticated to the CLI, for example by uploading an SSH public key to your Jenkins user account.

Then try:

java -jar /path/to/jenkins-cli.jar -s http://server/ support > ./path/to/support-bundle.zip

This command generates the support bundle and saves it to the path you provided.

Alternatively, if you are using the deprecated and insecure legacy remoting-based CLI, the generated support bundle is saved to a file in your default temporary folder using the filename printed.

You can also request only specific components to include; to see the currently available list, ask for the command’s help:

java -jar /path/to/jenkins-cli.jar -s http://server/ help support

Enabling anonymization

The diagnostic information collected can contain sensitive information, but this can be automatically filtered by enabling support bundle anonymization.

If support bundle anonymization is enabled through the Configure Global Security settings, some metadata is replaced with automatically generated anonymized names: nodes, computers, labels, users, items (including jobs), views, hostnames, and IP addresses. If you need to determine the real value for an anonymized one, you can look that up in the support bundle anonymization web page after logging into your product.

When anonymization is disabled, a warning message is shown on the Support web page.

WARNING: Support bundle anonymization is disabled. This can be enabled in the global configuration under Support Bundle Anonymization.

Click the link to Manage Jenkins  Configure Global Security and enable support bundle anonymization.

Anonymize support bundle contents checkbox
If you are using Jenkins Support Core plugin instead of the CloudBees Support plugin, use version 2.53 or higher of the Jenkins Support Core plugin due to the Known Performance issue with Support Bundle Anonymization.

Viewing Anonymized Mappings

When submitting an anonymized support bundle to your support organization, they may need to ask further details about items with anonymized names. To translate that, navigate to Manage Jenkins  Support Bundle Anonymization.

Support Bundle Anonymization management link

This page contains a table of mappings between original names and their corresponding anonymized versions. This also contains a list of stop words that are ignored when anonymization generates anonymized counterparts. These are common terms in Jenkins that by themselves convey no personal meaning. For example, an agent named "Jenkins" will not be anonymized because "jenkins" is a stop word.

Screenshot of anonymized mappings management page example

Anonymization limitations

Anonymization filters only apply to text files. It cannot handle non-Jenkins URLs, custom proprietary Jenkins plugin names, and exceptions quoting invalid Groovy code in a Jenkins pipeline. The active plugins, disabled plugins, failed plugins, and Dockerfile reports are not anonymized due to several Jenkins plugins and other Java libraries using version numbers that are indistinguishable from IP addresses. These reports are in the files plugins/active.txt, plugins/disabled.txt, plugins/failed.txt, and docker/Dockerfile. These files should all be manually reviewed if you do not wish to disclose the names of custom proprietary plugins.

Generating a support bundle when Jenkins will not start

When the CloudBees Support Plugin is installed, it automatically stores a bundle every hour in $JENKINS_HOME/support. These bundles are purged using an exponential retention strategy so that they do not overflow disk space.

If your Jenkins instance is not starting correctly, this is usually due to some class-loading conflict among the set of plugins that you have installed in your instance. Normally such class-loading conflicts just results in one of your plugins failing to load, however in some extreme cases a plugin failing to load can cause a second plugin to render the UI of your Jenkins instance inaccessible.

In such cases, the latest support bundle (and some representative historical versions) can be very helpful for CloudBees in ensuring a rapid response and restoration of your Jenkins to an accessible state.

Sending a support bundle too large for Zendesk

Files over 20 MB can’t be attached to a support ticket. If this is the case, you have 3 options:

  1. Recompress the file

  2. Use the CloudBees Uploads Service

  3. Split the file

These 3 options are explained in the next sections.

Recompress the file

Using an additional compression with a different algorithm like xz can help you to reduce the size of the archive. For example, on Linux/MacOS after having installed xz with the package manager you are using for your system (aptitude install xz, yum install xz, brew install xz,etc) you can use it like this:

xz -z -9 -e cloudbees-support.zip

CloudBees Uploads Service

Use https://uploads.cloudbees.com/ to upload the file to CloudBees Support.

Split the file

Another way is to split the archive file into separate smaller files and send them to us. The following command split the file mybundle.zip into several files named mybundle.zip.part-01, mybundle.zip.part-02, and so on and whose the size does not exceed 20 MB:

split --numeric-suffixes -b 20m mybundle.zip "mybundle.zip.part-"

To reassemble the split archive and access the files within:

cat mybundle.zip.part-* > mybundle.zip unzip -p mybundle.zip > mybundle.tar.xz mkdir mybundle tar -xf mybundle.tar.xz -C mybundle

Troubleshooting

The following sections are common problems you might encounter when generating a support bundle and how to resolve them.

The support bundle generated is missing 'pipeline-timings.txt'

If the support bundle is missing pipeline-timings.txt, the Jenkins logs shows an exception like the following:

Apr 23, 2018 2:19:06 PM WARNING com.cloudbees.jenkins.support.SupportPlugin writeBundle Could not attach 'nodes/master/pipeline-timings.txt' to support bundle java.io.IOException: <jobName> <buildNumber> did not yet start at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:1058) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$PipelineTimings$1.writeTo(CpsFlowExecution.java:1779) at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:357) at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:271) at com.cloudbees.jenkins.support.SupportPlugin$PeriodicWorkImpl$1.run(SupportPlugin.java:741) at java.lang.Thread.run(Thread.java:748)

This issue happens if pipeline timings are not available for a specific build at the time that the bundle is generated. This is causes by a non-graceful exception handing in the Pipeline Groovy plugin, that has been fixed in version 2.68 of this plugin.

Upgrade the Pipeline Groovy plugin to version 2.68 or later.

The support bundle generated is missing its Manifest

If the support bundle generated misses some files, you might see something similar to:

2019-07-02 17:06:31.129+0000 [id=847267] WARNING c.c.j.s.SupportPlugin$PeriodicWorkImpl#lambda$doRun$0: Could not save support bundle Command Close created at at hudson.remoting.Command.<init>(Command.java:68) at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1265) at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1263) at hudson.remoting.Channel.close(Channel.java:1436) at hudson.remoting.Channel.close(Channel.java:1403) at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1270) Caused: hudson.remoting.Channel$OrderlyShutdown at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1271) at hudson.remoting.Channel$1.handle(Channel.java:565) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:85) Caused: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on my-node failed. The channel is closing down or has closed down at hudson.remoting.Channel.call(Channel.java:948) at hudson.FilePath.act(FilePath.java:1072) at hudson.FilePath.act(FilePath.java:1061) at hudson.FilePath.lastModified(FilePath.java:1599) at com.cloudbees.jenkins.support.api.FilePathContent.getTime(FilePathContent.java:85) at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:301) at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:276) at com.cloudbees.jenkins.support.SupportPlugin$PeriodicWorkImpl.lambda$doRun$0(SupportPlugin.java:764) at com.cloudbees.jenkins.support.SupportPlugin$PeriodicWorkImpl.dt_access$313(SupportPlugin.java) at java.lang.Thread.run(Thread.java:748)

The support bundle generation may be interrupted by remoting issues while retrieving files on remote agents. When this happens, the bundle generation stops downloading more components but create the bundle nonetheless. This results in the support bundle being incomplete, missing a lot files, in particular its manifest.

The problem happens when getting files information on remote agents and could be avoided by disabling the following options in the Support page:

  • Agent JVM process system metrics (Linux only)

  • Agent system configuration (Linux only)

Upgrade the Jenkins Support Core plugin to version 2.58 or later to fix this issue.

Since enabling/turning on the Support bundle Anonymization feature, Jenkins has been running very slowly

This issue was caused by searching over the entire support bundle for sensitive information when there were many files which would never contain the information that needed to be redacted.

To resolve this issue, update to version 2.53 of the Jenkins Support Core plugin.

Capturing minimal setup information when a support bundle cannot be generated?

If the support bundle can not be generated, capture the minimal information as indicated below for the support team to help diagnose your issue.

Logs capture

You will need to navigate to your JENKINS_HOME and capture the content of:

  • $JENKINS_HOME/logs

  • $JENKINS_HOME/slow-requests

  • $JENKINS_HOME/deadlocks

Make an archive out of those three directories and attach it to the ticket. If the archive is larger than 20MB, please use this service to send it to us.

Instance detail capture

  1. Go to the Jenkins groovy console, under <jenkins_url>/script, and run:

println '# Overview' Jenkins.instance.with { println '## Instance' println "*Jenkins Version*: ${getVersion()}" println "*Java Version*: ${System.getProperty('java.version')}" Runtime.getRuntime().with { println "*Maximum memory*: ${humanReadableSize(maxMemory())}" println "*Allocated memory*: ${humanReadableSize(totalMemory())}" println "*Free memory*: ${humanReadableSize(freeMemory())}" } println "*Jenkins URL*: ${JenkinsLocationConfiguration.get().getUrl()}" println "*Instance ID*: ${getLegacyInstanceId()}" println '\n## Plugins' pluginManager.plugins.each { println("${it.getDisplayName()} (${it.getShortName()}): ${it.getVersion()}") } } println '\n\n\n# Properties' System.getProperties() def static humanReadableSize(long size) { def measure = 'B' if (size < 1024) { return "${size} ${measure}" } def number = size if (number >= 1024) { number = number / 1024 measure = 'KB' if (number >= 1024) { number = number / 1024 measure = 'MB' if (number >= 1024) { number = number / 1024 measure = 'GB' } } } return "${new java.text.DecimalFormat('#0.00').format(number)} ${measure} (${size})" }
  1. Copy the output into an overview.txt file and attach it to the ticket.

Thread dump capture

  1. Go to <jenkins_url>/threadDump and copy the content into a thread1.txt file.

  2. Wait a minute, refresh the page, and copy the new content into a thread2.txt file.

Monitoring data capture

This step depends on the JavaMelody plugin being installed. If the plugin is not installed, then skip this step.
  1. Go to <jenkins_url>/monitoring and take a screenshot of the page.

  2. Attach this screenshot to the ticket.

In August 2020, the Jenkins project voted to replace the term master with controller. We have taken a pragmatic approach to cleaning these up, ensuring the least amount of downstream impact as possible. CloudBees is committed to ensuring a culture and environment of inclusiveness and acceptance - this includes ensuring the changes are not just cosmetic ones, but pervasive. As this change happens, please note that the term master has been replaced through the latest versions of the CloudBees documentation with controller (as in managed controller, client controller, team controller) except when still used in the UI or in code.