Required Data: IO issues on Linux

Article ID:115003442371
3 minute readKnowledge base

Issue

  • I have found evidences of IO issues with the FileSystem where Jenkins Home is hosted.

Required Data IO issues

This article describes how to collect the minimum required information for troubleshooting IO issues.

If the required data is bigger than 50 MB you will not be able to use ZenDesk to upload all the information. On this case we would like to encourage you to use our upload service in order to attach all the required information.

Environment

Automatic Data collection

This is the preferred method if you are using a product supported by the cbsupport CLI. Current products supporting collecting performance data are:

Steps to follow are:

  1. Set up cbsupport

  2. Run cbsupport required-data io

  3. Collect the archive generated in the working directory of cbsupport and attach it to the ticket using our upload service

Manual Data collection - Required Data check list

Ideally, your environment is configured as documented in Prepare CloudBees CI for Support.

  • Support bundle

  • Output of the script testIO.sh

  • Description of the File System

  • Optional (if an I/O issue is suspected): output of the script listDProcessesNativeStacks.sh

Support Bundle

A support bundle from the problematic instance ideally when the issue is happening or in the worst case right after the issue is exposed. Please, follow the KB below in case you don’t know how to generate a support bundle.

Please, enable at minimum the following checks: controller system configuration (Linux only), System properties, controller Log Recorders, Slow Request Records, Deadlock Records, Thread dumps.

Find more details about the Support Bundle in What is a support-bundle and how to create one.

Output of the script testIO.sh

1. - Download the testIO.sh.

2. - Make the script executable

The testIO.sh script will need to be made executable by running: chmod +x testIO.sh.

3. - Get the $JENKINS_USER with ps ax | grep -i java | grep -i -e jenkins -e cloudbees | grep war

4. - Check that the jenkins user has permissions in the directory where testIO.sh is running

5. - Execute the testIO.sh script as the jenkins user by running:

sudo -u $JENKINS_USER sh testIO.sh $JENKINS_HOME [TIMES] [PERIOD]
▼

The testIO.sh script will execute Write and Write/Read test a number of [TIMES] times every [PERIOD].

For example:

sudo -u jenkins sh testIO.sh /var/lib/jenkins 2 15
▼

6. - Collect the <date>-testIO.txt generated and attach it to the ticket using our upload service

To find the expected values for this script, see: IO Troubleshooting on Linux.

Description of the File System

  • Output from df -H from the Host File System.

  • Output from cat /proc/mounts from the Host File System.

  • Let us know if JENKINS_HOME is mounted on a Share File System, please let us know its technology (NFS, etc.) and its version (NFSv4.1, etc.).

Output of the script listDProcessesNativeStacks.sh

See the What is listDProcessesNativeStacks.sh and how does it help? KB.