Selecting support-bundle options

Article ID:4402695625485
6 minute readKnowledge base

Issue

  • What support-bundle options should I select for X?

  • I am trying to troubleshoot X and CloudBees Support is asking me for a support bundle, which options should I select?

  • What do the support-bundle options mean?

support-bundle-options

Environment

Resolution

Agents

  • Agent Command Statistics

    Used to diagnose issues with agent to controller communication.

      $ cat [support bundle]/nodes/slave/[agent]/command-stats.md

  # Totals
  * Writes: 913
    * sent 2.4Mb
  * Reads: 1044
    * received 1.4Mb
  * Responses: 92
    * waited 9.1 sec

  # Commands sent
  * `Close`: 1
    * sent 0.0Mb
  * `Pipe.Chunk`: 5
    * sent 0.0Mb
  * `Pipe.Flush`: 2
    * sent 0.0Mb
  ...

  • Agent Log Recorder

    Captures agent logs.

      $ ls [support bundle]/nodes/slave/[agent]/logs/*
  $ cat [support bundle]/nodes/slave/[agent]/jenkins.log
  2021-02-20 01:33:03.955+0000 [id=172]   WARNING o.j.r.u.AnonymousClassWarnings#warn: Attempt to (de-)serialize anonymous class org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController$1; see: https://www.jenkins.io/redirect/serialization-of-anonymous-classes/
  2021-02-20 13:05:20.772+0000 [id=21622] INFO    h.r.RemoteInvocationHandler$Unexporter#reportStats: rate(1min) = {0,number,0.0}±{1,number,0.0}/sec; rate(5min) = {2,number,0.0}±{3,number,0.0}/sec; rate(15min) = {4,number,0.0}±{5,number,0.0}/sec; rate(total) = {6,number,0.0}±{7,number,0.0}/sec; N = {8,number}

  • Agents config file [Encrypted secrets are redacted]

    Agents configuration.

      $ cat [support bundle]/nodes/slave/[agent]/config.xml

  <slave>
    <name>agent-name</name>
    <remoteFS>/tmp/jenkins</remoteFS>
    <numExecutors>1</numExecutors>
    <mode>NORMAL</mode>
    <retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/>
    <launcher class="com.cloudbees.jenkins.plugins.sshslaves.SSHLauncher" plugin="cloudbees-ssh-slaves@2.7">
      <host>agent-name.example.com</host>
      <connectionDetails>
  ...

  • Agent JVM process system metrics (Linux only)

    Agent JVM details.

      ls [support bundle]/nodes/slave/[agent]/proc/*
  cpuinfo.txt		mounts.txt		self			system-uptime.txt
  meminfo.txt		net			swaps.txt

  • Agent system configuration (Linux only)

    General agent system information.

      $ ls [support bundle]/nodes/slave/[agent]/sysctl.txt
  $ ls [support bundle]/nodes/slave/[agent]/dmesg.txt
  $ ls [support bundle]/nodes/slave/[agent]/dmi.txt
  $ ls [support bundle]/nodes/slave/[agent]/environment.txt
  $ ls [support bundle]/nodes/slave/[agent]/file-descriptors.txt
  $ ls [support bundle]/nodes/slave/[agent]/sysctl.txt
  $ ls [support bundle]/nodes/slave/[agent]/proc/*

  • Dump agent export tables (could reveal some memory leaks) Note: agent memory leaks

    Agent export tables.

      $ cat [support bundle]/nodes/slave/[agent]/exportTable.txt
  #1 (ref.1) : object=hudson.remoting.Channel@57234fcd:jnk2stl30 type=hudson.remoting.Channel interfaces=[hudson.remoting.IChannel]
    Created at Fri Feb 19 19:14:54 CST 2021
  	at hudson.remoting.ExportTable$Entry.<init>(ExportTable.java:114)
  	at hudson.remoting.ExportTable.export(ExportTable.java:365)
  	at hudson.remoting.Channel.internalExport(Channel.java:821)
  ...

Client

  • About user (basic authentication details only)

    User details.

      User
  ====

  Authentication
  --------------

    * Authenticated: true
    * Name: GROUP\UserID
    * Authorities
        - `synchronized`
        - `authenticated`
  ...

CloudBees

  • CloudBees Assurance Program

    Beekeeper and Beekeeper-related properties.

      $ cat [support bundle]/cap/beekeeper.md

  # Configuration

  WAR Envelope detected [cje-2.249.2.4]
  Installed Envelope detected [cje-2.249.2.4]
  CloudBees Assurance Program enabled: false
  Allow automatic upgrades of individual plugins on startup: false
  Allow automatic downgrades of individual plugins on startup: false
  Notify when security warnings affecting the core are detected: true
  Notify when security warnings affecting plugins are detected: true

  • Operations Center Connector Logs

    Logs recorded on controller for operation center connection.

      $ ls [support bundle]/operations-center-connector-logs/*
  connection-details.md			operations-center-connector.log.3
  operations-center-connector.log		operations-center-connector.log.4
  operations-center-connector.log.1	operations-center-connector.log.5
  operations-center-connector.log.2

Jenkins

  • About Jenkins

    Fundamental Jenkins details.

      $ cat [support bundle]/about.md

  Jenkins
  =======

  Version details
  ---------------

    * Version: `2.263.4.2`
    * Instance ID: `0123456789012345678901234`
    * Mode:    Webapp Directory
    * Url:     https://jenkins.example.com/cjoc/
  ...

  • All loggers currently enabled.

    Enabled loggers.

      $ cat [support bundle]/loggers.md
  Loggers currently enabled
  =========================
  org.apache.sshd - WARNING
  org.jenkinsci.plugins.gwt - ALL
  disabled - OFF
  com.splunk.splunkjenkins.HealthMonitor - WARNING
  winstone - INFO
   - INFO

  • Build queue

      $ cat [support bundle]/buildqueue.md
  Current build queue has 0 item(s).
  ---------------
  Is quieting down: false

  • Jenkins Global Configuration File (Encrypted secrets are redacted)

    Main configuration file ($JENKINS_HOME/config.xml)

      $ cat [support bundle]/jenkins-root-configuration-files/config.xml
  <hudson>
    <disabledAdministrativeMonitors>
      <string>hudson.diagnosis.TooManyJobsButNoView</string>
      <string>com.cloudbees.jenkins.plugins.assurance.SecurityWarningsWatch</string>
    </disabledAdministrativeMonitors>
    <version>2.235.1.2</version>
    <installStateName>RUNNING</installStateName>
    <numExecutors>0</numExecutors>
    <mode>NORMAL</mode>
    <useSecurity>true</useSecurity>
  ...

  • Items Content (Computationally expensive)

    Summary of jobs and types.

      Item statistics
  ===============

    * `hudson.model.FreeStyleProject`
      - Number of items: 35
      - Number of builds per job: 20.085714285714285 [n=35, s=40.0]
    * `org.jenkinsci.plugins.workflow.job.WorkflowJob`
      - Number of items: 20
      - Number of builds per job: 11.9 [n=20, s=20.0]

  Total job statistics
  ======================

    * Number of jobs: 55
    * Number of builds per job: 17.10909090909091 [n=55, s=40.0]
  ...

  • controller Log Recorders

    Custom log recorders (master/controller)

      $ ls [support bundle]/nodes/master/logs/custom/*

  • controller system configuration (Linux only)

    Specific controller configuration for your instance.

      $ ls [support bundle]/nodes/master/proc/*
  $ ls [support bundle]/nodes/master/systctl.txt
  $ ls [support bundle]/nodes/master/dmesg.txt
  $ ls [support bundle]/nodes/master/userid.txt
  $ ls [support bundle]/nodes/master/dmi.txt
  ...

  • Metrics

    Collection of metrics in json format.

      $ cat [support bundle]/nodes/master/metrics.json
  {
    "version" : "4.0.0",
    "gauges" : {
      "jenkins.executor.count.value" : {
        "value" : 41
      },
      "jenkins.executor.free.value" : {
        "value" : 36
      },
      "jenkins.executor.in-use.value" : {
        "value" : 5
  ...

  • Node monitors

      $ cat [support bundle]/node-monitors.md
  Node monitors
  =============
  Approved Folders
  ----
   - Is Ignored: false
   - Computers:
     * controller: null
     * agent01: null
     * agent02: null
     * agent03: null
  ...

  • Other Jenkins Configuration File (Encrypted secrets are redacted)

    The rest of the configuration files in the $JENKINS_HOME directory.

      $ ls [support bundle]/jenkins-root-configuration-files/
  io.jenkins.plugins.casc.CasCGlobalConfig.xml
  audit-trail.xml
  esr-feeder.xml
  jenkins.metrics.api.MetricsAccessKey.xml
  ...

  • Administrative monitors

    Built-in monitors.

      $ cat [support bundle]/admin-monitors.md
  Monitors
  ========

  `AsyncResourceDisposer`
  --------------
  (active and enabled)

  `jenkins.diagnostics.URICheckEncodingMonitor`
  --------------
  (active and enabled)

  • Update Center

    Update Center details.

      === Sites ===
   - Url: file:/tmp/jenkins_home/WEB-INF/plugins/update-center.json
   - Connection Url: null
   - Implementation Type: hudson.model.UpdateSite
   - Url: https://jenkins-updates.cloudbees.com/update-center/envelope-cje/update-center.json
   - Connection Url: https://www.google.com/
   - Implementation Type: hudson.model.UpdateSite
  ======
  Last updated: N/A
  === Proxy ===

JVM/Server

  • Environment variables

    Environment for the JVM process of your instance.

      $ cat [support bundle]/nodes/master/environment.txt
  HOME=/var/lib/jenkins
  LANG=en_US.UTF-8
  LOGNAME=jenkins
  PATH=/sbin:/usr/sbin:/bin:/usr/bin
  PWD=/
  ...

  • File descriptors (Unix only)

    Open files of your instance.

      $ cat [support bundle]/nodes/master/file-descriptors.txt
  Jenkins
  ======

  core file size          (blocks, -c) 0
  data seg size           (kbytes, -d) unlimited
  scheduling priority             (-e) 0
  file size               (blocks, -f) unlimited
  pending signals                 (-i) 123756
  max locked memory       (kbytes, -l) 64
  max memory size         (kbytes, -m) unlimited
  open files                      (-n) 65534
  pipe size            (512 bytes, -p) 8
  ...

  • controller Heap Histogram

    A heap histogram if enabled.

      $ cat [support bundle]/nodes/master/heap-histogram.txt
  Histogram generation is disabled. If you want to enable it, do either:
  * Add the system property: -Dcom.cloudbees.jenkins.support.impl.HeapUsageHistogram.DISABLED=false
  * Run from Script Console the line: com.cloudbees.jenkins.support.impl.HeapUsageHistogram.DISABLED=false

  • controller JVM process system metrics (Linux only)

      $ ls [support bundle]/nodes/master/proc/meminfo.txt
  $ ls [support bundle]/nodes/master/proc/self/*
  cpuinfo.txt		mounts.txt		self			system-uptime.txt
  meminfo.txt		net			swaps.txt

  • Network Interface

      $ cat [support bundle]/nodes/master/networkInterface.md
  -----------
   * Name eth0
   ** Hardware Address - abba3f5d6003
   ** Index - 3
   ** InetAddress - /fe80:0:0:0:d0af:1fff:fe5d:6003%eth0
   ** InetAddress - /10.4.1.9
   ** MTU - 1500
   ** Is Up - true
   ** Is Virtual - false
   ** Is Loopback - false
   ** Is Point to Point - false
   ** Supports multicast - true

  • Reverse Proxy

      $ cat [support bundle]/reverse-proxy.md
  Reverse Proxy
  =============
   * Detected `X-Forwarded-For` header: TRUE

  • Root CAs

      $ cat [support bundle]/nodes/master/RootCA.txt
  ===== Trust Manager 0 =====
  It is an X.509 Trust Manager containing 204 certificates:
  CN=ITF MC Production Network Applications sub CA, OU=Global Information Security, O=controllerCard WorldWide, DC=controllercard, DC=com
  CN=SecureTrust CA, O=SecureTrust Corporation, C=US
  CN=Entrust Root Certification Authority - EC1, OU="(c) 2012 Entrust, Inc. - for authorized use only", OU=See www.entrust.net/legal-terms, O="Entrust, Inc.", C=US
  CN=DigiCert Global Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US
  OU=Security Communication RootCA1, O=SECOM Trust.net, C=JP
  ...
  $ cat [support bundle]/slave/[agent]/RootCA.txt
  ===== Trust Manager 0 =====
  It is an X.509 Trust Manager containing 133 certificates:
  CN=Hongkong Post Root CA 1, O=Hongkong Post, C=HK
  CN=SecureTrust CA, O=SecureTrust Corporation, C=US
  CN=Entrust Root Certification Authority - EC1, OU="(c) 2012 Entrust, Inc. - for authorized use only", OU=See www.entrust.net/legal-terms, O="Entrust, Inc.", C=US
  ...

  • System properties

    JVM system properties (i.e. java -XshowSettings)

      $ cat [support bundle]/nodes/master/system.properties
  #Fri May 01 14:31:28 UTC 2020
  MASTER_DOMAIN=domain
  MASTER_ENDPOINT=https\://example.com/devops/
  MASTER_GRANT_ID=14a45fff-3985-eeee-977f-aa8f4e0e1425
  MASTER_INDEX=1
  ...
  $ cat [support bundle]/nodes/slave/[agent]/system.properties
  #Fri Aug 21 03:51:33 CST 2020
  awt.toolkit=sun.awt.X11.XToolkit
  file.encoding=UTF-8
  file.encoding.pkg=sun.io
  file.separator=/
  ...

Performance

  • Deadlock Records

    Detected deadlocks.

      $ ls [support bundle]/deadlocks/*
  DeadlockDetected-20200320-185437.txt

  • Garbage Collection Logs

      $ ls [support bundle]/nodes/master/logs/gc/*
  gc.log.1		gc.log.2.current

  • Load Statistics

      $ ls [support bundle]/load-stats/
  label		no-label	overall
  $ ls [support bundle]/load-stats/label/
  service-f8b099dd-7777-9999-98b0-3599e73a21bc
  java-8
  master
  $ ls [support bundle]/load-stats/label/master/
  gnuplot		hour.csv	min.csv		sec10.csv

  • Out of Memory Errors

      $ cat [support bundle]/nodes/master/oome.md
  OutOfMemoryError Statistics
  ===========================

    * Time of first OOME: N/A

  • Slow Requests Records

    URI requests which responded slowly.

      $ ls [support bundle]/slow-requests/
  20210225-023235.323.txt	20210225-095644.322.txt	20210225-155605.322.txt	20210225-164326.323.txt
  20210225-023235.324.txt	20210225-140117.322.txt	20210225-155929.322.txt	20210225-164929.322.txt
  20210225-064626.322.txt	20210225-141029.322.txt	20210225-160032.322.txt	20210225-165253.322.txt
  20210225-064629.322.txt	20210225-143444.322.txt	20210225-160111.322.txt	20210225-165332.322.txt
  ...

  • Thread dumps

    Snapshot of all Jenkins threads and their stacktraces (Note: critical performance data)

       $ cat [support bundle]/nodes/master/thread-dump.txt
   Master
   ======

   "Analytics" id=1540 (0x604) state=TIMED_WAITING cpu=17%
       - waiting on <0x58430729> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
       - locked <0x58430729> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
       at sun.misc.Unsafe.park(Native Method)
       at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2083)
   ...

  • Thread dumps of running Pipeline builds

    Currently running pipelines and their thread dumps.

      $ cat [support bundle]/nodes/master/pipeline-thread-dump.txt
  Build: [Job] #1
  Thread #0
          at WorkflowScript.run(WorkflowScript:30)
  Approximate graph size: 4

  Build: [Job]/dynatrace mapping #3
  Thread #0
          at org.[organization].pipeline.workflow.PCFWorkflow.createJsonStageList(PCFWorkflow.groovy:82)
          at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2030)
          at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2015)
          at com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:2056)
          at org.[organization].pipeline.workflow.PCFWorkflow.createJsonStageList(PCFWorkflow.groovy:81)
          at my.call(my.groovy:67)
          at WorkflowScript.run(WorkflowScript:2)
  Approximate graph size: 21
  ...

  • Timing data about recently completed Pipeline builds

    Pipeline build timings.

      $ cat [support bundle]/nodes/master/pipeline-timings.txt
  No timings available for [Folder]/[Job]/dev #1: java.io.IOException: [Folder]/[Job]/dev #1 did not yet start

  No timings available for [Folder]/[Job]/master #1: java.io.IOException: [Folder]/[Job]/master #1 did not yet start
  ...

Plugins

  • About Jenkins

    Global information about the instance.

      $ cat [support bundle]/about.md

  • Plugins

    A list of active plugins (plugins/active.txt), disable plugins (plugins/disabled.txt), failed plugins (plugins/failed.txt and backed up plugins (plugins/backup.txt).

User Interface (UI)

  • About browser

    Your browser.

      $ cat [support bundle]/browser.md
  Browser
  =======

    * Screen size: 2048x1152
    * User Agent
        - Type:     Browser
        - Name:     Chrome
        - Family:   CHROME
        - Producer: Google Inc.
        - Version:  88.0.4324.96
        - Raw:      `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.96 Safari/537.36`
    * Operating System
        - Name:     OS X
        - Family:   OS_X
        - Producer: Apple Computer, Inc.
        - Version:  10.13.6

References

This article is part of our Knowledge Base and is provided for guidance-based purposes only. The solutions or workarounds described here are not officially supported by CloudBees and may not be applicable in all environments. Use at your own discretion, and test changes in a safe environment before applying them to production systems.