Issue
-
What support-bundle options should I select for X?
-
I am trying to troubleshoot X and CloudBees Support is asking me for a support bundle, which options should I select?
-
What do the support-bundle options mean?
Resolution
Agents
-
Agent Command Statistics
Used to diagnose issues with agent to controller communication.
$ cat [support bundle]/nodes/slave/[agent]/command-stats.md # Totals * Writes: 913 * sent 2.4Mb * Reads: 1044 * received 1.4Mb * Responses: 92 * waited 9.1 sec # Commands sent * `Close`: 1 * sent 0.0Mb * `Pipe.Chunk`: 5 * sent 0.0Mb * `Pipe.Flush`: 2 * sent 0.0Mb ...
-
Agent Log Recorder
Captures agent logs.
$ ls [support bundle]/nodes/slave/[agent]/logs/* $ cat [support bundle]/nodes/slave/[agent]/jenkins.log 2021-02-20 01:33:03.955+0000 [id=172] WARNING o.j.r.u.AnonymousClassWarnings#warn: Attempt to (de-)serialize anonymous class org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController$1; see: https://www.jenkins.io/redirect/serialization-of-anonymous-classes/ 2021-02-20 13:05:20.772+0000 [id=21622] INFO h.r.RemoteInvocationHandler$Unexporter#reportStats: rate(1min) = {0,number,0.0}±{1,number,0.0}/sec; rate(5min) = {2,number,0.0}±{3,number,0.0}/sec; rate(15min) = {4,number,0.0}±{5,number,0.0}/sec; rate(total) = {6,number,0.0}±{7,number,0.0}/sec; N = {8,number}
-
Agents config file [Encrypted secrets are redacted]
Agents configuration.
$ cat [support bundle]/nodes/slave/[agent]/config.xml <slave> <name>agent-name</name> <remoteFS>/tmp/jenkins</remoteFS> <numExecutors>1</numExecutors> <mode>NORMAL</mode> <retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/> <launcher class="com.cloudbees.jenkins.plugins.sshslaves.SSHLauncher" plugin="cloudbees-ssh-slaves@2.7"> <host>agent-name.example.com</host> <connectionDetails> ...
-
Agent JVM process system metrics (Linux only)
Agent JVM details.
ls [support bundle]/nodes/slave/[agent]/proc/* cpuinfo.txt mounts.txt self system-uptime.txt meminfo.txt net swaps.txt
-
Agent system configuration (Linux only)
General agent system information.
$ ls [support bundle]/nodes/slave/[agent]/sysctl.txt $ ls [support bundle]/nodes/slave/[agent]/dmesg.txt $ ls [support bundle]/nodes/slave/[agent]/dmi.txt $ ls [support bundle]/nodes/slave/[agent]/environment.txt $ ls [support bundle]/nodes/slave/[agent]/file-descriptors.txt $ ls [support bundle]/nodes/slave/[agent]/sysctl.txt $ ls [support bundle]/nodes/slave/[agent]/proc/*
-
Dump agent export tables (could reveal some memory leaks) Note: agent memory leaks
Agent export tables.
$ cat [support bundle]/nodes/slave/[agent]/exportTable.txt #1 (ref.1) : object=hudson.remoting.Channel@57234fcd:jnk2stl30 type=hudson.remoting.Channel interfaces=[hudson.remoting.IChannel] Created at Fri Feb 19 19:14:54 CST 2021 at hudson.remoting.ExportTable$Entry.<init>(ExportTable.java:114) at hudson.remoting.ExportTable.export(ExportTable.java:365) at hudson.remoting.Channel.internalExport(Channel.java:821) ...
Client
-
About user (basic authentication details only)
User details.
User ==== Authentication -------------- * Authenticated: true * Name: GROUP\UserID * Authorities - `synchronized` - `authenticated` ...
CloudBees
-
CloudBees Assurance Program
Beekeeper and Beekeeper-related properties.
$ cat [support bundle]/cap/beekeeper.md # Configuration WAR Envelope detected [cje-2.249.2.4] Installed Envelope detected [cje-2.249.2.4] CloudBees Assurance Program enabled: false Allow automatic upgrades of individual plugins on startup: false Allow automatic downgrades of individual plugins on startup: false Notify when security warnings affecting the core are detected: true Notify when security warnings affecting plugins are detected: true
-
Operations Center Connector Logs
Logs recorded on controller for operation center connection.
$ ls [support bundle]/operations-center-connector-logs/* connection-details.md operations-center-connector.log.3 operations-center-connector.log operations-center-connector.log.4 operations-center-connector.log.1 operations-center-connector.log.5 operations-center-connector.log.2
Jenkins
-
About Jenkins
Fundamental Jenkins details.
$ cat [support bundle]/about.md Jenkins ======= Version details --------------- * Version: `2.263.4.2` * Instance ID: `0123456789012345678901234` * Mode: Webapp Directory * Url: https://jenkins.example.com/cjoc/ ...
-
All loggers currently enabled.
Enabled loggers.
$ cat [support bundle]/loggers.md Loggers currently enabled ========================= org.apache.sshd - WARNING org.jenkinsci.plugins.gwt - ALL disabled - OFF com.splunk.splunkjenkins.HealthMonitor - WARNING winstone - INFO - INFO
-
Build queue
$ cat [support bundle]/buildqueue.md Current build queue has 0 item(s). --------------- Is quieting down: false
-
Jenkins Global Configuration File (Encrypted secrets are redacted)
Main configuration file (
$JENKINS_HOME
/config.xml)$ cat [support bundle]/jenkins-root-configuration-files/config.xml <hudson> <disabledAdministrativeMonitors> <string>hudson.diagnosis.TooManyJobsButNoView</string> <string>com.cloudbees.jenkins.plugins.assurance.SecurityWarningsWatch</string> </disabledAdministrativeMonitors> <version>2.235.1.2</version> <installStateName>RUNNING</installStateName> <numExecutors>0</numExecutors> <mode>NORMAL</mode> <useSecurity>true</useSecurity> ...
-
Items Content (Computationally expensive)
Summary of jobs and types.
Item statistics =============== * `hudson.model.FreeStyleProject` - Number of items: 35 - Number of builds per job: 20.085714285714285 [n=35, s=40.0] * `org.jenkinsci.plugins.workflow.job.WorkflowJob` - Number of items: 20 - Number of builds per job: 11.9 [n=20, s=20.0] Total job statistics ====================== * Number of jobs: 55 * Number of builds per job: 17.10909090909091 [n=55, s=40.0] ...
-
controller Log Recorders
Custom log recorders (master/controller)
$ ls [support bundle]/nodes/master/logs/custom/*
-
controller system configuration (Linux only)
Specific controller configuration for your instance.
$ ls [support bundle]/nodes/master/proc/* $ ls [support bundle]/nodes/master/systctl.txt $ ls [support bundle]/nodes/master/dmesg.txt $ ls [support bundle]/nodes/master/userid.txt $ ls [support bundle]/nodes/master/dmi.txt ...
-
Metrics
Collection of metrics in json format.
$ cat [support bundle]/nodes/master/metrics.json { "version" : "4.0.0", "gauges" : { "jenkins.executor.count.value" : { "value" : 41 }, "jenkins.executor.free.value" : { "value" : 36 }, "jenkins.executor.in-use.value" : { "value" : 5 ...
-
Node monitors
$ cat [support bundle]/node-monitors.md Node monitors ============= Approved Folders ---- - Is Ignored: false - Computers: * controller: null * agent01: null * agent02: null * agent03: null ...
-
Other Jenkins Configuration File (Encrypted secrets are redacted)
The rest of the configuration files in the
$JENKINS_HOME
directory.$ ls [support bundle]/jenkins-root-configuration-files/ io.jenkins.plugins.casc.CasCGlobalConfig.xml audit-trail.xml esr-feeder.xml jenkins.metrics.api.MetricsAccessKey.xml ...
-
Administrative monitors
Built-in monitors.
$ cat [support bundle]/admin-monitors.md Monitors ======== `AsyncResourceDisposer` -------------- (active and enabled) `jenkins.diagnostics.URICheckEncodingMonitor` -------------- (active and enabled)
-
Update Center
Update Center details.
=== Sites === - Url: file:/tmp/jenkins_home/WEB-INF/plugins/update-center.json - Connection Url: null - Implementation Type: hudson.model.UpdateSite - Url: https://jenkins-updates.cloudbees.com/update-center/envelope-cje/update-center.json - Connection Url: https://www.google.com/ - Implementation Type: hudson.model.UpdateSite ====== Last updated: N/A === Proxy ===
JVM/Server
-
Environment variables
Environment for the JVM process of your instance.
$ cat [support bundle]/nodes/master/environment.txt HOME=/var/lib/jenkins LANG=en_US.UTF-8 LOGNAME=jenkins PATH=/sbin:/usr/sbin:/bin:/usr/bin PWD=/ ...
-
File descriptors (Unix only)
Open files of your instance.
$ cat [support bundle]/nodes/master/file-descriptors.txt Jenkins ====== core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 123756 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 65534 pipe size (512 bytes, -p) 8 ...
-
controller Heap Histogram
A heap histogram if enabled.
$ cat [support bundle]/nodes/master/heap-histogram.txt Histogram generation is disabled. If you want to enable it, do either: * Add the system property: -Dcom.cloudbees.jenkins.support.impl.HeapUsageHistogram.DISABLED=false * Run from Script Console the line: com.cloudbees.jenkins.support.impl.HeapUsageHistogram.DISABLED=false
-
controller JVM process system metrics (Linux only)
$ ls [support bundle]/nodes/master/proc/meminfo.txt $ ls [support bundle]/nodes/master/proc/self/* cpuinfo.txt mounts.txt self system-uptime.txt meminfo.txt net swaps.txt
-
Network Interface
$ cat [support bundle]/nodes/master/networkInterface.md ----------- * Name eth0 ** Hardware Address - abba3f5d6003 ** Index - 3 ** InetAddress - /fe80:0:0:0:d0af:1fff:fe5d:6003%eth0 ** InetAddress - /10.4.1.9 ** MTU - 1500 ** Is Up - true ** Is Virtual - false ** Is Loopback - false ** Is Point to Point - false ** Supports multicast - true
-
Reverse Proxy
$ cat [support bundle]/reverse-proxy.md Reverse Proxy ============= * Detected `X-Forwarded-For` header: TRUE
-
Root CAs
$ cat [support bundle]/nodes/master/RootCA.txt ===== Trust Manager 0 ===== It is an X.509 Trust Manager containing 204 certificates: CN=ITF MC Production Network Applications sub CA, OU=Global Information Security, O=controllerCard WorldWide, DC=controllercard, DC=com CN=SecureTrust CA, O=SecureTrust Corporation, C=US CN=Entrust Root Certification Authority - EC1, OU="(c) 2012 Entrust, Inc. - for authorized use only", OU=See www.entrust.net/legal-terms, O="Entrust, Inc.", C=US CN=DigiCert Global Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US OU=Security Communication RootCA1, O=SECOM Trust.net, C=JP ... $ cat [support bundle]/slave/[agent]/RootCA.txt ===== Trust Manager 0 ===== It is an X.509 Trust Manager containing 133 certificates: CN=Hongkong Post Root CA 1, O=Hongkong Post, C=HK CN=SecureTrust CA, O=SecureTrust Corporation, C=US CN=Entrust Root Certification Authority - EC1, OU="(c) 2012 Entrust, Inc. - for authorized use only", OU=See www.entrust.net/legal-terms, O="Entrust, Inc.", C=US ...
-
System properties
JVM system properties (i.e. java -XshowSettings)
$ cat [support bundle]/nodes/master/system.properties #Fri May 01 14:31:28 UTC 2020 MASTER_DOMAIN=domain MASTER_ENDPOINT=https\://example.com/devops/ MASTER_GRANT_ID=14a45fff-3985-eeee-977f-aa8f4e0e1425 MASTER_INDEX=1 ... $ cat [support bundle]/nodes/slave/[agent]/system.properties #Fri Aug 21 03:51:33 CST 2020 awt.toolkit=sun.awt.X11.XToolkit file.encoding=UTF-8 file.encoding.pkg=sun.io file.separator=/ ...
Performance
-
Deadlock Records
Detected deadlocks.
$ ls [support bundle]/deadlocks/* DeadlockDetected-20200320-185437.txt
-
Garbage Collection Logs
$ ls [support bundle]/nodes/master/logs/gc/* gc.log.1 gc.log.2.current
-
Load Statistics
$ ls [support bundle]/load-stats/ label no-label overall $ ls [support bundle]/load-stats/label/ service-f8b099dd-7777-9999-98b0-3599e73a21bc java-8 master $ ls [support bundle]/load-stats/label/master/ gnuplot hour.csv min.csv sec10.csv
-
Out of Memory Errors
$ cat [support bundle]/nodes/master/oome.md OutOfMemoryError Statistics =========================== * Time of first OOME: N/A
-
Slow Requests Records
URI requests which responded slowly.
$ ls [support bundle]/slow-requests/ 20210225-023235.323.txt 20210225-095644.322.txt 20210225-155605.322.txt 20210225-164326.323.txt 20210225-023235.324.txt 20210225-140117.322.txt 20210225-155929.322.txt 20210225-164929.322.txt 20210225-064626.322.txt 20210225-141029.322.txt 20210225-160032.322.txt 20210225-165253.322.txt 20210225-064629.322.txt 20210225-143444.322.txt 20210225-160111.322.txt 20210225-165332.322.txt ...
-
Thread dumps
Snapshot of all Jenkins threads and their stacktraces (Note: critical performance data)
$ cat [support bundle]/nodes/master/thread-dump.txt Master ====== "Analytics" id=1540 (0x604) state=TIMED_WAITING cpu=17% - waiting on <0x58430729> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x58430729> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2083) ...
-
Thread dumps of running Pipeline builds
Currently running pipelines and their thread dumps.
$ cat [support bundle]/nodes/master/pipeline-thread-dump.txt Build: [Job] #1 Thread #0 at WorkflowScript.run(WorkflowScript:30) Approximate graph size: 4 Build: [Job]/dynatrace mapping #3 Thread #0 at org.[organization].pipeline.workflow.PCFWorkflow.createJsonStageList(PCFWorkflow.groovy:82) at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2030) at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2015) at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2056) at org.[organization].pipeline.workflow.PCFWorkflow.createJsonStageList(PCFWorkflow.groovy:81) at my.call(my.groovy:67) at WorkflowScript.run(WorkflowScript:2) Approximate graph size: 21 ...
-
Timing data about recently completed Pipeline builds
Pipeline build timings.
$ cat [support bundle]/nodes/master/pipeline-timings.txt No timings available for [Folder]/[Job]/dev #1: java.io.IOException: [Folder]/[Job]/dev #1 did not yet start No timings available for [Folder]/[Job]/master #1: java.io.IOException: [Folder]/[Job]/master #1 did not yet start ...
Plugins
-
About Jenkins
Global information about the instance.
$ cat [support bundle]/about.md
-
Plugins
A list of active plugins (
plugins/active.txt
), disable plugins (plugins/disabled.txt
), failed plugins (plugins/failed.txt
and backed up plugins (plugins/backup.txt
).
User Interface (UI)
-
About browser
Your browser.
$ cat [support bundle]/browser.md Browser ======= * Screen size: 2048x1152 * User Agent - Type: Browser - Name: Chrome - Family: CHROME - Producer: Google Inc. - Version: 88.0.4324.96 - Raw: `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.96 Safari/537.36` * Operating System - Name: OS X - Family: OS_X - Producer: Apple Computer, Inc. - Version: 10.13.6