CloudBees Performance Decision Tree for troubleshooting

This performance troubleshooting decision tree is intended to help you clarify and isolate the root causes for common performance problems with Jenkins-based products, including CloudBees Core, CloudBees Jenkins Distribution, CloudBees Jenkins Platform, CloudBees Jenkins Enterprise v1, and Jenkins LTS.

Is your production Jenkins instance down?

Is your production Jenkins instance unavailable or unusable because of performance issues?

Is application navigation slow or unresponsive?

Is your production Jenkins instance available and usable, but the navigation is very slow or unresponsive?

Are you experiencing high CPU or memory consumption?

Is your production Jenkins instance available and usable, but you’re seeing high CPU or memory consumption?

Are jobs executing very slowly?

Is your production Jenkins instance available and usable, but jobs are executing very slowly or not at all?

Are you experiencing an issue with a particular job?

Is your production Jenkins instance available and usable, but you’re experiencing issues with a specific job?

My problem isn’t listed here

If you’re experiencing a problem not already listed here, submit a Support Request explaining the situation in as much detail as possible. Note that you may be asked to provide additional data for analysis.

Subsystem performance troubleshooting

This section provides additional troubleshooting guidance for specific instance subsystems.

Reviewing the Java version and JVM arguments

Are you using best practices, including using the G1 garbage collector?

If you’re seeing issues with Java and/or the JVM, review JVM troubleshooting, and check to make sure that your system is using the G1 garbage collector.

Is the JDK a minimum of 1.8.0_212?

Check your system to make sure that the JDK in use is at least version 1.8.0_212 or higher. JDKs earlier than 1.8.0_212 are not supported.

The supported JDKs for each product are listed in the Supported Platforms pages:

Analyzing thread dumps

Thread dumps are found in the Support Bundle archive as thread-dump.txt files under the /nodes/ directory in the folder for each node. For example, if you have a node called master, you can find thread dumps for that node under /nodes/master.

Once you’ve collected thread dumps for your instance, use the free fastthread.io resource to analyze the thread dumps:

  1. In a browser window, navigate to https://fastthread.io.

  2. In the Upload thread dump section, click Choose file to select the thread dump file.

  3. CLick Analyze to upload the thread dumps for analysis.

After uploading and analysis, you will be presented with a report that details findings from the thread dump analysis.

Is the thread count near 700?

Contact CloudBees Support for additional troubleshooting guidance.

Are there a lot of blocked threads?

Contact CloudBees Support for additional troubleshooting guidance.

Are there a lot of threads with the same stack trace?

Contact CloudBees Support for additional troubleshooting guidance.

Analyzing garbage collection logs

Garbage collection logs can be captured as part of a Support Bundle. To capture garbage collection logs:

  1. Click on Support.

  2. Select Garbage Collection Logs.

  3. Click Generate Bundle.

Once you have captured garbage collection logs, use the free gceasy.io resource analyze those logs:

  1. In a browser window, navigate to https://gceasy.io.

  2. In the Upload GC Log File section, click Choose file to select the garbage collection logs.

  3. Click Analyze to upload the logs for analysis.

After uploading and analysis, you will be presented with a report that details any garbage collection issues discovered.

Is application throughput less than 90%?

Contact CloudBees Support for additional troubleshooting guidance.

Are there garbage collection pauses of greater than 1 second?

Contact CloudBees Support for additional troubleshooting guidance.

Do the heap graphs indicate a memory leak?

Contact CloudBees Support for additional troubleshooting guidance.

Is metaspace usage increasing?

Contact CloudBees Support for additional troubleshooting guidance.

Reviewing testio.sh output

The testio.sh script tests input/output throughput for your instance. For usage instructions, see the Knowledge Base article Required Data: IO issues on Linux.

If the testio.sh output indicates issues, you may be using an outdated or unsupported version of NFS. For guidance on using NFS, see the NFS Guide.

Copyright © 2010-2020 CloudBees, Inc.Online version published by CloudBees, Inc. under the Creative Commons Attribution-ShareAlike 4.0 license.CloudBees and CloudBees DevOptics are registered trademarks and CloudBees Core, CloudBees Flow, CloudBees Flow Deploy, CloudBees Flow DevOps Insight, CloudBees Flow DevOps Foresight, CloudBees Flow Release, CloudBees Accelerator, CloudBees Accelerator ElectricInsight, CloudBees Accelerator Electric Make, CloudBees CodeShip, CloudBees Jenkins Enterprise, CloudBees Jenkins Platform, CloudBees Jenkins Operations Center, and DEV@cloud are trademarks of CloudBees, Inc. Most CloudBees products are commonly referred to by their short names — Accelerator, Automation Platform, Flow, Deploy, Foresight, Release, Insight, and eMake — throughout various types of CloudBees product-specific documentation. Oracle and Java are registered trademarks of Oracle and/or its affiliates. Jenkins is a registered trademark of the non-profit Software in the Public Interest organization. Used with permission. See here for more info about the Jenkins project. The registered trademark Jenkins® is used pursuant to a sublicense from the Jenkins project and Software in the Public Interest, Inc. Read more at www.cloudbees.com/jenkins/about. Apache, Apache Ant, Apache Maven, Ant and Maven are trademarks of The Apache Software Foundation. Used with permission. No endorsement by The Apache Software Foundation is implied by the use of these marks.Other names may be trademarks of their respective owners. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this content, and CloudBees was aware of a trademark claim, the designations have been printed in caps or initial caps. While every precaution has been taken in the preparation of this content, the publisher and authors assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein.