The CloudBees Support plugin provides the ability to generate a bundle of all the commonly-requested diagnostic information used by CloudBees when resolving support issues. It extends the open source Support Core plugin, which is installed as a dependency when the CloudBees Support plugin is installed.
The CloudBees Support plugin is installed by default with CloudBees CI.
This guide explains what a support bundle is and how to generate, both from the UI and the CLI.
What is a support bundle
A bundle is a simple .zip file containing mostly plain text files from the root of your Jenkins home directory, $JENKINS_HOME, which can be used to assist a CloudBees Support Engineer in diagnosing your issue. You can inspect the contents of this file to assure yourself that it does not contain information you do not want to share with CloudBees. You can even unpack and repack the bundle if there is some specific piece of information that you need to remove from the bundle.
What a support bundle includes
In summary, a support bundle contains:
-
IP addresses (yes, but see Enabling anonymization)
-
Hostnames
-
CIDR (same as IP addresses)
The following are not included in a support bundle:
-
Credentials
-
Usernames (no, except
jenkinsuser) -
Job Details
-
Groovy Pipeline Code
An example filename of a support bundle is cloudbees-support_core-cm-controller.example.com_2022-11-09_22.55.52.zip.
The following is an example of the zip file contents:
├── about.md (<-- Hostname) ├── admin-monitors.md ├── agent-protocols.md ├── browser.md ├── buildqueue.md ├── cap │ ├── beekeeper.md │ └── properties.md ├── docker │ └── Dockerfile ├── identity.md ├── items.md ├── load-stats │ ├── label │ │ └── label_dark_researcher │ │ ├── gnuplot │ │ ├── hour.csv │ │ ├── min.csv │ │ └── sec10.csv │ ├── no-label │ │ ├── gnuplot │ │ ├── hour.csv │ │ ├── min.csv │ │ └── sec10.csv │ └── overall │ ├── gnuplot │ ├── hour.csv │ ├── min.csv │ └── sec10.csv ├── loggers.md ├── manifest.md ├── node-monitors.md ├── nodes │ └── master │ ├── checksums.md5 │ ├── dmesg.txt │ ├── dmi.txt │ ├── environment.txt │ ├── file-descriptors.txt │ ├── logs (<-- Hostname, IP Addresses) │ │ ├── all_2022-11-09_22.32.49.log │ │ ├── all_memory_buffer.log │ │ └── jenkins.log │ ├── metrics.json │ ├── networkInterface.md (<-- IP Address) │ ├── oome.md │ ├── pipeline-thread-dump.txt │ ├── pipeline-timings.txt │ ├── proc │ │ ├── cpuinfo.txt │ │ ├── meminfo.txt │ │ ├── mounts.txt │ │ ├── net │ │ │ └── rpc │ │ │ ├── nfs.txt │ │ │ └── nfsd.txt │ │ ├── self │ │ │ ├── cmdline (<-- Hostname) │ │ │ ├── environ (<-- Hostname, Username) │ │ │ ├── limits.txt │ │ │ ├── mountstats.txt │ │ │ └── status.txt │ │ ├── swaps.txt │ │ └── system-uptime.txt │ ├── sysctl.txt │ ├── system.properties │ ├── thread-dump.txt │ └── userid.txt (<-- Username) ├── nodes.md ├── operations-center-connector-logs │ ├── connection-details.md │ ├── operations-center-connector.log ├── other-logs │ ├── copy_reference_file.log ├── plugins │ ├── active.txt │ ├── backup.txt │ ├── disabled.txt │ └── failed.txt ├── reverse-proxy.md ├── running-builds.txt ├── task-logs │ ├── com.cloudbees.opscenter.client.cloud.CloudImpl$PeriodicWorkImpl.log │ ├── health-checker.log │ ├── operations-center-connector.log │ ├── SharedConfigurationSynchronizer.log ├── update-center.md ├── user.md └── users └── count.md 16 directories, 78 files
Best practices when generating a support bundle
To provide you a better support service, please consider the following recommendations:
-
Review steps described in Prepare CloudBees CI for Support.
-
“About Jenkins” is mandatory.
-
The closer the Bundle generation is to the issue, the better for CloudBees Support to diagnose it.
-
In case you are sending a series of support bundles, please attach them separately (in
.zipformat, by default). Do not compress them into a unique package. -
If your support bundle has a big file size, you could “recompress” a zip with xz. See Sending a support bundle too large for Zendesk for details.
-
If you are experiencing an issue regarding the operations center - client controller connectivity problem, send support bundles for each of the instances.
-
To avoid noise on logs, indicate on the ticket the date and time when the particular issue happened/arose.
-
If you can reproduce the issue by yourself, indicate the followed steps before generating the support bundle.
Generating a support bundle from the UI
To generate a support bundle, navigate to the operations center or controller dashboard, and then select Support.
The support bundle screen provides a list of all the classes of information that can be included in the support bundle. Normally it is best to include all the selected information, however there may be some information which you do not want to share for reasons of confidentiality. In such cases you can de-select the information you do not want to include in the bundle.
Click the Generate Bundle to download the bundle to your machine. A
bundle is a simple .zip file containing mostly plain text files. You
can inspect the contents of this file to assure yourself that it does
not contain information you do not want to share with CloudBees. You can
even unpack and repack the bundle if there is some specific piece of
information that you need to remove from the bundle.
When you are happy with the bundle, attach the bundle to your CloudBees support ticket.
Generating a support bundle from the CLI
If there is a problem accessing the web UI, of course it is going to be difficult to diagnose those by going to that same UI and getting a bundle. As an alternative, you can use the Jenkins CLI to obtain a bundle from a shell.
Before you encounter problems, make sure you have downloaded jenkins-cli.jar from the server (go to /cli/ to get a link). You may also need to make sure you have authenticated to the CLI, for example by uploading an SSH public key to your Jenkins user account.
Then try:
java -jar /path/to/jenkins-cli.jar -s http://server/ support > ./path/to/support-bundle.zip
This command generates the support bundle and saves it to the path you provided.
Alternatively, if you are using the deprecated and insecure legacy remoting-based CLI, the generated support bundle is saved to a file in your default temporary folder using the filename printed.
You can also request only specific components to include; to see the currently available list, ask for the command’s help:
java -jar /path/to/jenkins-cli.jar -s http://server/ help support
Enabling anonymization
The diagnostic information collected can contain sensitive information, but this can be automatically filtered by enabling support bundle anonymization.
If support bundle anonymization is enabled through the Configure Global Security settings, some metadata is replaced with automatically generated anonymized names: nodes, computers, labels, users, items (including jobs), views, hostnames, and IP addresses. If you need to determine the real value for an anonymized one, you can look that up in the support bundle anonymization web page after logging into your product.
When anonymization is disabled, a warning message is shown on the Support web page.
Click the link to and enable support bundle anonymization.
| If you are using Jenkins Support Core plugin instead of the CloudBees Support plugin, use version 2.53 or higher of the Jenkins Support Core plugin due to the Known Performance issue with Support Bundle Anonymization. |
Viewing Anonymized Mappings
When submitting an anonymized support bundle to your support organization, they may need to ask further details about items with anonymized names. To translate that, navigate to .
This page contains a table of mappings between original names and their corresponding anonymized versions. This also contains a list of stop words that are ignored when anonymization generates anonymized counterparts. These are common terms in Jenkins that by themselves convey no personal meaning. For example, an agent named "Jenkins" will not be anonymized because "jenkins" is a stop word.
Anonymization limitations
Anonymization filters only apply to text files.
It cannot handle non-Jenkins URLs, custom proprietary Jenkins plugin names, and exceptions quoting invalid Groovy code in a Jenkins pipeline.
The active plugins, disabled plugins, failed plugins, and Dockerfile reports are not anonymized due to several Jenkins plugins and other Java libraries using version numbers that are indistinguishable from IP addresses.
These reports are in the files plugins/active.txt, plugins/disabled.txt, plugins/failed.txt, and docker/Dockerfile.
These files should all be manually reviewed if you do not wish to disclose the names of custom proprietary plugins.
Generating a support bundle when Jenkins will not start
When the CloudBees Support Plugin is installed, it automatically
stores a bundle every hour in $JENKINS_HOME/support. These
bundles are purged using an exponential retention strategy so that they
do not overflow disk space.
If your Jenkins instance is not starting correctly, this is usually due to some class-loading conflict among the set of plugins that you have installed in your instance. Normally such class-loading conflicts just results in one of your plugins failing to load, however in some extreme cases a plugin failing to load can cause a second plugin to render the UI of your Jenkins instance inaccessible.
In such cases, the latest support bundle (and some representative historical versions) can be very helpful for CloudBees in ensuring a rapid response and restoration of your Jenkins to an accessible state.
Sending a support bundle too large for Zendesk
Files over 20 MB can’t be attached to a support ticket. If this is the case, you have 3 options:
-
Recompress the file
-
Use the CloudBees Uploads Service
-
Split the file
These 3 options are explained in the next sections.
Recompress the file
Using an additional compression with a different algorithm like xz can help you to reduce the size of the archive. For example, on Linux/MacOS after having installed xz with the package manager you are using for your system (aptitude install xz, yum install xz, brew install xz,etc) you can use it like this:
xz -z -9 -e cloudbees-support.zip
CloudBees Uploads Service
Use https://uploads.cloudbees.com/ to upload the file to CloudBees Support.
Split the file
Another way is to split the archive file into separate smaller files and send them to us. The following command split the file mybundle.zip into several files named mybundle.zip.part-01, mybundle.zip.part-02, and so on and whose the size does not exceed 20 MB:
split --numeric-suffixes -b 20m mybundle.zip "mybundle.zip.part-"
To reassemble the split archive and access the files within:
cat mybundle.zip.part-* > mybundle.zip unzip -p mybundle.zip > mybundle.tar.xz mkdir mybundle tar -xf mybundle.tar.xz -C mybundle
Troubleshooting
The following sections are common problems you might encounter when generating a support bundle and how to resolve them.
The support bundle generated is missing 'pipeline-timings.txt'
If the support bundle is missing pipeline-timings.txt, the Jenkins logs shows an exception like the following:
Apr 23, 2018 2:19:06 PM WARNING com.cloudbees.jenkins.support.SupportPlugin writeBundle Could not attach 'nodes/master/pipeline-timings.txt' to support bundle java.io.IOException: <jobName> <buildNumber> did not yet start at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:1058) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$PipelineTimings$1.writeTo(CpsFlowExecution.java:1779) at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:357) at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:271) at com.cloudbees.jenkins.support.SupportPlugin$PeriodicWorkImpl$1.run(SupportPlugin.java:741) at java.lang.Thread.run(Thread.java:748)
This issue happens if pipeline timings are not available for a specific build at the time that the bundle is generated. This is causes by a non-graceful exception handing in the Pipeline Groovy plugin, that has been fixed in version 2.68 of this plugin.
Upgrade the Pipeline Groovy plugin to version 2.68 or later.
The support bundle generated is missing its Manifest
If the support bundle generated misses some files, you might see something similar to:
2019-07-02 17:06:31.129+0000 [id=847267] WARNING c.c.j.s.SupportPlugin$PeriodicWorkImpl#lambda$doRun$0: Could not save support bundle Command Close created at at hudson.remoting.Command.<init>(Command.java:68) at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1265) at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1263) at hudson.remoting.Channel.close(Channel.java:1436) at hudson.remoting.Channel.close(Channel.java:1403) at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1270) Caused: hudson.remoting.Channel$OrderlyShutdown at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1271) at hudson.remoting.Channel$1.handle(Channel.java:565) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:85) Caused: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on my-node failed. The channel is closing down or has closed down at hudson.remoting.Channel.call(Channel.java:948) at hudson.FilePath.act(FilePath.java:1072) at hudson.FilePath.act(FilePath.java:1061) at hudson.FilePath.lastModified(FilePath.java:1599) at com.cloudbees.jenkins.support.api.FilePathContent.getTime(FilePathContent.java:85) at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:301) at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:276) at com.cloudbees.jenkins.support.SupportPlugin$PeriodicWorkImpl.lambda$doRun$0(SupportPlugin.java:764) at com.cloudbees.jenkins.support.SupportPlugin$PeriodicWorkImpl.dt_access$313(SupportPlugin.java) at java.lang.Thread.run(Thread.java:748)
The support bundle generation may be interrupted by remoting issues while retrieving files on remote agents. When this happens, the bundle generation stops downloading more components but create the bundle nonetheless. This results in the support bundle being incomplete, missing a lot files, in particular its manifest.
The problem happens when getting files information on remote agents and could be avoided by disabling the following options in the Support page:
-
Agent JVM process system metrics (Linux only)
-
Agent system configuration (Linux only)
Upgrade the Jenkins Support Core plugin to version 2.58 or later to fix this issue.
Since enabling/turning on the Support bundle Anonymization feature, Jenkins has been running very slowly
This issue was caused by searching over the entire support bundle for sensitive information when there were many files which would never contain the information that needed to be redacted.
To resolve this issue, update to version 2.53 of the Jenkins Support Core plugin.
Capturing minimal setup information when a support bundle cannot be generated?
If the support bundle can not be generated, capture the minimal information as indicated below for the support team to help diagnose your issue.
Logs capture
You will need to navigate to your JENKINS_HOME and capture the content of:
-
$JENKINS_HOME/logs -
$JENKINS_HOME/slow-requests -
$JENKINS_HOME/deadlocks
Make an archive out of those three directories and attach it to the ticket. If the archive is larger than 20MB, please use this service to send it to us.
Instance detail capture
-
Go to the Jenkins groovy console, under
<jenkins_url>/script, and run:
println '# Overview' Jenkins.instance.with { println '## Instance' println "*Jenkins Version*: ${getVersion()}" println "*Java Version*: ${System.getProperty('java.version')}" Runtime.getRuntime().with { println "*Maximum memory*: ${humanReadableSize(maxMemory())}" println "*Allocated memory*: ${humanReadableSize(totalMemory())}" println "*Free memory*: ${humanReadableSize(freeMemory())}" } println "*Jenkins URL*: ${JenkinsLocationConfiguration.get().getUrl()}" println "*Instance ID*: ${getLegacyInstanceId()}" println '\n## Plugins' pluginManager.plugins.each { println("${it.getDisplayName()} (${it.getShortName()}): ${it.getVersion()}") } } println '\n\n\n# Properties' System.getProperties() def static humanReadableSize(long size) { def measure = 'B' if (size < 1024) { return "${size} ${measure}" } def number = size if (number >= 1024) { number = number / 1024 measure = 'KB' if (number >= 1024) { number = number / 1024 measure = 'MB' if (number >= 1024) { number = number / 1024 measure = 'GB' } } } return "${new java.text.DecimalFormat('#0.00').format(number)} ${measure} (${size})" }
-
Copy the output into an
overview.txtfile and attach it to the ticket.
Thread dump capture
-
Go to
<jenkins_url>/threadDumpand copy the content into athread1.txtfile. -
Wait a minute, refresh the page, and copy the new content into a
thread2.txtfile.
Monitoring data capture
| This step depends on the JavaMelody plugin being installed. If the plugin is not installed, then skip this step. |
-
Go to
<jenkins_url>/monitoringand take a screenshot of the page. -
Attach this screenshot to the ticket.