Selecting support-bundle options

Article ID:4402695625485
6 minute readKnowledge base

Issue

  • What support-bundle options should I select for X?

  • I am trying to troubleshoot X and CloudBees Support is asking me for a support bundle, which options should I select?

  • What do the support-bundle options mean?

support-bundle-options

Environment

Resolution

Agents

  • Agent Command Statistics

    Used to diagnose issues with agent to controller communication.

      $ cat [support bundle]/nodes/slave/[agent]/command-stats.md

  # Totals
  * Writes: 913
    * sent 2.4Mb
  * Reads: 1044
    * received 1.4Mb
  * Responses: 92
    * waited 9.1 sec

  # Commands sent
  * `Close`: 1
    * sent 0.0Mb
  * `Pipe.Chunk`: 5
    * sent 0.0Mb
  * `Pipe.Flush`: 2
    * sent 0.0Mb
  ...

  • Agent Log Recorder

    Captures agent logs.

      $ ls [support bundle]/nodes/slave/[agent]/logs/*
  $ cat [support bundle]/nodes/slave/[agent]/jenkins.log
  2021-02-20 01:33:03.955+0000 [id=172]   WARNING o.j.r.u.AnonymousClassWarnings#warn: Attempt to (de-)serialize anonymous class org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController$1; see: https://www.jenkins.io/redirect/serialization-of-anonymous-classes/
  2021-02-20 13:05:20.772+0000 [id=21622] INFO    h.r.RemoteInvocationHandler$Unexporter#reportStats: rate(1min) = {0,number,0.0}±{1,number,0.0}/sec; rate(5min) = {2,number,0.0}±{3,number,0.0}/sec; rate(15min) = {4,number,0.0}±{5,number,0.0}/sec; rate(total) = {6,number,0.0}±{7,number,0.0}/sec; N = {8,number}

  • Agents config file [Encrypted secrets are redacted]

    Agents configuration.

      $ cat [support bundle]/nodes/slave/[agent]/config.xml

  <slave>
    <name>agent-name</name>
    <remoteFS>/tmp/jenkins</remoteFS>
    <numExecutors>1</numExecutors>
    <mode>NORMAL</mode>
    <retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/>
    <launcher class="com.cloudbees.jenkins.plugins.sshslaves.SSHLauncher" plugin="cloudbees-ssh-slaves@2.7">
      <host>agent-name.example.com</host>
      <connectionDetails>
  ...

  • Agent JVM process system metrics (Linux only)

    Agent JVM details.

      ls [support bundle]/nodes/slave/[agent]/proc/*
  cpuinfo.txt		mounts.txt		self			system-uptime.txt
  meminfo.txt		net			swaps.txt

  • Agent system configuration (Linux only)

    General agent system information.

      $ ls [support bundle]/nodes/slave/[agent]/sysctl.txt
  $ ls [support bundle]/nodes/slave/[agent]/dmesg.txt
  $ ls [support bundle]/nodes/slave/[agent]/dmi.txt
  $ ls [support bundle]/nodes/slave/[agent]/environment.txt
  $ ls [support bundle]/nodes/slave/[agent]/file-descriptors.txt
  $ ls [support bundle]/nodes/slave/[agent]/sysctl.txt
  $ ls [support bundle]/nodes/slave/[agent]/proc/*

  • Dump agent export tables (could reveal some memory leaks) Note: agent memory leaks

    Agent export tables.

      $ cat [support bundle]/nodes/slave/[agent]/exportTable.txt
  #1 (ref.1) : object=hudson.remoting.Channel@57234fcd:jnk2stl30 type=hudson.remoting.Channel interfaces=[hudson.remoting.IChannel]
    Created at Fri Feb 19 19:14:54 CST 2021
  	at hudson.remoting.ExportTable$Entry.<init>(ExportTable.java:114)
  	at hudson.remoting.ExportTable.export(ExportTable.java:365)
  	at hudson.remoting.Channel.internalExport(Channel.java:821)
  ...

Client

  • About user (basic authentication details only)

    User details.

      User
  ====

  Authentication
  --------------

    * Authenticated: true
    * Name: GROUP\UserID
    * Authorities
        - `synchronized`
        - `authenticated`
  ...

CloudBees

  • CloudBees Assurance Program

    Beekeeper and Beekeeper-related properties.

      $ cat [support bundle]/cap/beekeeper.md

  # Configuration

  WAR Envelope detected [cje-2.249.2.4]
  Installed Envelope detected [cje-2.249.2.4]
  CloudBees Assurance Program enabled: false
  Allow automatic upgrades of individual plugins on startup: false
  Allow automatic downgrades of individual plugins on startup: false
  Notify when security warnings affecting the core are detected: true
  Notify when security warnings affecting plugins are detected: true

  • Operations Center Connector Logs

    Logs recorded on controller for operation center connection.

      $ ls [support bundle]/operations-center-connector-logs/*
  connection-details.md			operations-center-connector.log.3
  operations-center-connector.log		operations-center-connector.log.4
  operations-center-connector.log.1	operations-center-connector.log.5
  operations-center-connector.log.2

Jenkins

  • About Jenkins

    Fundamental Jenkins details.

      $ cat [support bundle]/about.md

  Jenkins
  =======

  Version details
  ---------------

    * Version: `2.263.4.2`
    * Instance ID: `0123456789012345678901234`
    * Mode:    Webapp Directory
    * Url:     https://jenkins.example.com/cjoc/
  ...

  • All loggers currently enabled.

    Enabled loggers.

      $ cat [support bundle]/loggers.md
  Loggers currently enabled
  =========================
  org.apache.sshd - WARNING
  org.jenkinsci.plugins.gwt - ALL
  disabled - OFF
  com.splunk.splunkjenkins.HealthMonitor - WARNING
  winstone - INFO
   - INFO

  • Build queue

      $ cat [support bundle]/buildqueue.md
  Current build queue has 0 item(s).
  ---------------
  Is quieting down: false

  • Jenkins Global Configuration File (Encrypted secrets are redacted)

    Main configuration file ($JENKINS_HOME/config.xml)

      $ cat [support bundle]/jenkins-root-configuration-files/config.xml
  <hudson>
    <disabledAdministrativeMonitors>
      <string>hudson.diagnosis.TooManyJobsButNoView</string>
      <string>com.cloudbees.jenkins.plugins.assurance.SecurityWarningsWatch</string>
    </disabledAdministrativeMonitors>
    <version>2.235.1.2</version>
    <installStateName>RUNNING</installStateName>
    <numExecutors>0</numExecutors>
    <mode>NORMAL</mode>
    <useSecurity>true</useSecurity>
  ...

  • Items Content (Computationally expensive)

    Summary of jobs and types.

      Item statistics
  ===============

    * `hudson.model.FreeStyleProject`
      - Number of items: 35
      - Number of builds per job: 20.085714285714285 [n=35, s=40.0]
    * `org.jenkinsci.plugins.workflow.job.WorkflowJob`
      - Number of items: 20
      - Number of builds per job: 11.9 [n=20, s=20.0]

  Total job statistics
  ======================

    * Number of jobs: 55
    * Number of builds per job: 17.10909090909091 [n=55, s=40.0]
  ...

  • controller Log Recorders

    Custom log recorders (master/controller)

      $ ls [support bundle]/nodes/master/logs/custom/*

  • controller system configuration (Linux only)

    Specific controller configuration for your instance.

      $ ls [support bundle]/nodes/master/proc/*
  $ ls [support bundle]/nodes/master/systctl.txt
  $ ls [support bundle]/nodes/master/dmesg.txt
  $ ls [support bundle]/nodes/master/userid.txt
  $ ls [support bundle]/nodes/master/dmi.txt
  ...

  • Metrics

    Collection of metrics in json format.

      $ cat [support bundle]/nodes/master/metrics.json
  {
    "version" : "4.0.0",
    "gauges" : {
      "jenkins.executor.count.value" : {
        "value" : 41
      },
      "jenkins.executor.free.value" : {
        "value" : 36
      },
      "jenkins.executor.in-use.value" : {
        "value" : 5
  ...

  • Node monitors

      $ cat [support bundle]/node-monitors.md
  Node monitors
  =============
  Approved Folders
  ----
   - Is Ignored: false
   - Computers:
     * controller: null
     * agent01: null
     * agent02: null
     * agent03: null
  ...

  • Other Jenkins Configuration File (Encrypted secrets are redacted)

    The rest of the configuration files in the $JENKINS_HOME directory.

      $ ls [support bundle]/jenkins-root-configuration-files/
  io.jenkins.plugins.casc.CasCGlobalConfig.xml
  audit-trail.xml
  esr-feeder.xml
  jenkins.metrics.api.MetricsAccessKey.xml
  ...

  • Administrative monitors

    Built-in monitors.

      $ cat [support bundle]/admin-monitors.md
  Monitors
  ========

  `AsyncResourceDisposer`
  --------------
  (active and enabled)

  `jenkins.diagnostics.URICheckEncodingMonitor`
  --------------
  (active and enabled)

  • Update Center

    Update Center details.

      === Sites ===
   - Url: file:/tmp/jenkins_home/WEB-INF/plugins/update-center.json
   - Connection Url: null
   - Implementation Type: hudson.model.UpdateSite
   - Url: https://jenkins-updates.cloudbees.com/update-center/envelope-cje/update-center.json
   - Connection Url: https://www.google.com/
   - Implementation Type: hudson.model.UpdateSite
  ======
  Last updated: N/A
  === Proxy ===

JVM/Server

  • Environment variables

    Environment for the JVM process of your instance.

      $ cat [support bundle]/nodes/master/environment.txt
  HOME=/var/lib/jenkins
  LANG=en_US.UTF-8
  LOGNAME=jenkins
  PATH=/sbin:/usr/sbin:/bin:/usr/bin
  PWD=/
  ...

  • File descriptors (Unix only)

    Open files of your instance.

      $ cat [support bundle]/nodes/master/file-descriptors.txt
  Jenkins
  ======

  core file size          (blocks, -c) 0
  data seg size           (kbytes, -d) unlimited
  scheduling priority             (-e) 0
  file size               (blocks, -f) unlimited
  pending signals                 (-i) 123756
  max locked memory       (kbytes, -l) 64
  max memory size         (kbytes, -m) unlimited
  open files                      (-n) 65534
  pipe size            (512 bytes, -p) 8
  ...

  • controller Heap Histogram

    A heap histogram if enabled.

      $ cat [support bundle]/nodes/master/heap-histogram.txt
  Histogram generation is disabled. If you want to enable it, do either:
  * Add the system property: -Dcom.cloudbees.jenkins.support.impl.HeapUsageHistogram.DISABLED=false
  * Run from Script Console the line: com.cloudbees.jenkins.support.impl.HeapUsageHistogram.DISABLED=false

  • controller JVM process system metrics (Linux only)

      $ ls [support bundle]/nodes/master/proc/meminfo.txt
  $ ls [support bundle]/nodes/master/proc/self/*
  cpuinfo.txt		mounts.txt		self			system-uptime.txt
  meminfo.txt		net			swaps.txt

  • Network Interface

      $ cat [support bundle]/nodes/master/networkInterface.md
  -----------
   * Name eth0
   ** Hardware Address - abba3f5d6003
   ** Index - 3
   ** InetAddress - /fe80:0:0:0:d0af:1fff:fe5d:6003%eth0
   ** InetAddress - /10.4.1.9
   ** MTU - 1500
   ** Is Up - true
   ** Is Virtual - false
   ** Is Loopback - false
   ** Is Point to Point - false
   ** Supports multicast - true

  • Reverse Proxy

      $ cat [support bundle]/reverse-proxy.md
  Reverse Proxy
  =============
   * Detected `X-Forwarded-For` header: TRUE

  • Root CAs

      $ cat [support bundle]/nodes/master/RootCA.txt
  ===== Trust Manager 0 =====
  It is an X.509 Trust Manager containing 204 certificates:
  CN=ITF MC Production Network Applications sub CA, OU=Global Information Security, O=controllerCard WorldWide, DC=controllercard, DC=com
  CN=SecureTrust CA, O=SecureTrust Corporation, C=US
  CN=Entrust Root Certification Authority - EC1, OU="(c) 2012 Entrust, Inc. - for authorized use only", OU=See www.entrust.net/legal-terms, O="Entrust, Inc.", C=US
  CN=DigiCert Global Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US
  OU=Security Communication RootCA1, O=SECOM Trust.net, C=JP
  ...
  $ cat [support bundle]/slave/[agent]/RootCA.txt
  ===== Trust Manager 0 =====
  It is an X.509 Trust Manager containing 133 certificates:
  CN=Hongkong Post Root CA 1, O=Hongkong Post, C=HK
  CN=SecureTrust CA, O=SecureTrust Corporation, C=US
  CN=Entrust Root Certification Authority - EC1, OU="(c) 2012 Entrust, Inc. - for authorized use only", OU=See www.entrust.net/legal-terms, O="Entrust, Inc.", C=US
  ...

  • System properties

    JVM system properties (i.e. java -XshowSettings)

      $ cat [support bundle]/nodes/master/system.properties
  #Fri May 01 14:31:28 UTC 2020
  MASTER_DOMAIN=domain
  MASTER_ENDPOINT=https\://example.com/devops/
  MASTER_GRANT_ID=14a45fff-3985-eeee-977f-aa8f4e0e1425
  MASTER_INDEX=1
  ...
  $ cat [support bundle]/nodes/slave/[agent]/system.properties
  #Fri Aug 21 03:51:33 CST 2020
  awt.toolkit=sun.awt.X11.XToolkit
  file.encoding=UTF-8
  file.encoding.pkg=sun.io
  file.separator=/
  ...

Performance

  • Deadlock Records

    Detected deadlocks.

      $ ls [support bundle]/deadlocks/*
  DeadlockDetected-20200320-185437.txt

  • Garbage Collection Logs

      $ ls [support bundle]/nodes/master/logs/gc/*
  gc.log.1		gc.log.2.current

  • Load Statistics

      $ ls [support bundle]/load-stats/
  label		no-label	overall
  $ ls [support bundle]/load-stats/label/
  service-f8b099dd-7777-9999-98b0-3599e73a21bc
  java-8
  master
  $ ls [support bundle]/load-stats/label/master/
  gnuplot		hour.csv	min.csv		sec10.csv

  • Out of Memory Errors

      $ cat [support bundle]/nodes/master/oome.md
  OutOfMemoryError Statistics
  ===========================

    * Time of first OOME: N/A

  • Slow Requests Records

    URI requests which responded slowly.

      $ ls [support bundle]/slow-requests/
  20210225-023235.323.txt	20210225-095644.322.txt	20210225-155605.322.txt	20210225-164326.323.txt
  20210225-023235.324.txt	20210225-140117.322.txt	20210225-155929.322.txt	20210225-164929.322.txt
  20210225-064626.322.txt	20210225-141029.322.txt	20210225-160032.322.txt	20210225-165253.322.txt
  20210225-064629.322.txt	20210225-143444.322.txt	20210225-160111.322.txt	20210225-165332.322.txt
  ...

  • Thread dumps

    Snapshot of all Jenkins threads and their stacktraces (Note: critical performance data)

       $ cat [support bundle]/nodes/master/thread-dump.txt
   Master
   ======

   "Analytics" id=1540 (0x604) state=TIMED_WAITING cpu=17%
       - waiting on <0x58430729> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
       - locked <0x58430729> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
       at sun.misc.Unsafe.park(Native Method)
       at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2083)
   ...

  • Thread dumps of running Pipeline builds

    Currently running pipelines and their thread dumps.

      $ cat [support bundle]/nodes/master/pipeline-thread-dump.txt
  Build: [Job] #1
  Thread #0
          at WorkflowScript.run(WorkflowScript:30)
  Approximate graph size: 4

  Build: [Job]/dynatrace mapping #3
  Thread #0
          at org.[organization].pipeline.workflow.PCFWorkflow.createJsonStageList(PCFWorkflow.groovy:82)
          at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2030)
          at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2015)
          at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2056)
          at org.[organization].pipeline.workflow.PCFWorkflow.createJsonStageList(PCFWorkflow.groovy:81)
          at my.call(my.groovy:67)
          at WorkflowScript.run(WorkflowScript:2)
  Approximate graph size: 21
  ...

  • Timing data about recently completed Pipeline builds

    Pipeline build timings.

      $ cat [support bundle]/nodes/master/pipeline-timings.txt
  No timings available for [Folder]/[Job]/dev #1: java.io.IOException: [Folder]/[Job]/dev #1 did not yet start

  No timings available for [Folder]/[Job]/master #1: java.io.IOException: [Folder]/[Job]/master #1 did not yet start
  ...

Plugins

  • About Jenkins

    Global information about the instance.

      $ cat [support bundle]/about.md

  • Plugins

    A list of active plugins (plugins/active.txt), disable plugins (plugins/disabled.txt), failed plugins (plugins/failed.txt and backed up plugins (plugins/backup.txt).

User Interface (UI)

  • About browser

    Your browser.

      $ cat [support bundle]/browser.md
  Browser
  =======

    * Screen size: 2048x1152
    * User Agent
        - Type:     Browser
        - Name:     Chrome
        - Family:   CHROME
        - Producer: Google Inc.
        - Version:  88.0.4324.96
        - Raw:      `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.96 Safari/537.36`
    * Operating System
        - Name:     OS X
        - Family:   OS_X
        - Producer: Apple Computer, Inc.
        - Version:  10.13.6

References