Configure Loggers for Jenkins and CloudBees CI

Issue

I want to adjust the log levels of Jenkins or CloudBees CI for specific packages/classes to troubleshoot an issue.

Environment

Resolution

There are different ways of configuring the logging levels of a controller or operations center. The most common way is to use the UI to add a logger, but if the UI is not accessible, multiple alternatives are listed below.

Changing the log level of a package can generate a lot of logs and can affect the performance of your controller due to increased IO throughput (causing iowait). It is recommended to change the log level of a package only when you are troubleshooting an issue and to change it back to the default level when you finish.

Solution 1: Adding a logger from the UI (recommended)

Go to Manage Jenkins System Log Add recorder (previously Add New Log Recorder)
Provide a name for the logger (does not have to be the package/class name)
Under the Loggers section, click on Add to add a new logger
Add the package/class name you want to monitor (this field will auto-complete based on the classes available in the controller)
Choose the log level
Click on Save

To share the logs with CloudBees Support, generate a support bundle including the Controller Custom Log Recorders option.

Solution 2: use a post initialization script:

Jenkins provides a way to run groovy script during initialization: Post-initialization script. So you can create a script that set the logging levels appropriately on startup. For example, the following script set the level of hudson.security.csrf.CrumbFilter and hudson.plugins.git.GitStatus to the SEVERE level:

import java.util.logging.Level
import java.util.logging.Logger

Logger.getLogger("hudson.plugins.git.GitStatus").setLevel(Level.SEVERE)
Logger.getLogger("hudson.security.csrf.CrumbFilter").setLevel(Level.SEVERE)

Solution 3: use `java.util.logging`

Create a file logging.properties in which you define the logging levels and a ConsoleHandler. Then pass this file to the JVM by adding the system property -Djava.util.logging.config.file=<pathTo>/logging.properties. A file like the following would apply the same configuration as in Solution 1:

.level = INFO
handlers= java.util.logging.ConsoleHandler

java.util.logging.ConsoleHandler.level=INFO
java.util.logging.ConsoleHandler.formatter=java.util.logging.SimpleFormatter

hudson.security.csrf.CrumbFilter.level = SEVERE
hudson.plugins.git.GitStatus.level = SEVERE

See the Java Logging Overview for more details about this method.

(Note: This solution works for classes that use java.util.logging. For classes that use a different implementation - for example org.apache.commons.logging - the level will not be changed)

Solution 4: Adding a logger via the filesystem JENKINS_HOME/log/*.xml

Pre-requisite: Support Core plugin. is installed. This plugin will write custom log recorders into $JENKINS_HOME/logs/custom/ on the Jenkins controller’s filesystem. You can then tail the relevant file to see updates to it. (Jenkins 2.114 introduced a system property which allows you to specify a different location to store Jenkins logs. Support Core plugin 2.47 supports this, so you may need to check if that property has been set if you are not seeing logs in the default location.)

$JENKINS_HOME (root)
+- log
|   +- kb-article.xml         # Custom log recorder definition
+- logs
   +- custom
      +- kb-article.log       # Custom log recorder file▼

You can add a logger by creating a new xml file under ${JENKINS_HOME}/log.

The folder is named log, not logs.

The UI logger name (e.g. <name>kb-article</name>) and the name of the file must keep consistency (e.g. kb-article.xml) because it is the way the System log recorder maps the custom logger under $JENKINS_HOME/logs/custom.

Here is an example creating a logger called kb-article for the com.cloudbees package on all kind of levels:

cat >kb-article.xml <<"EOF"
  <?xml version='1.1' encoding='UTF-8'?>
    <log>
      <name>kb-article</name>
      <targets>
        <target>
          <name>com.cloudbees</name>
          <level>-2147483648</level> <!--all -->
        </target>
        <target>
          <name>com.cloudbees</name>
          <level>300</level> <!--finest -->
        </target>
        <target>
          <name>com.cloudbees</name>
          <level>400</level> <!--finer -->
        </target>
        <target>
          <name>com.cloudbees</name>
          <level>500</level> <!--fine -->
        </target>
        <target>
          <name>com.cloudbees</name>
          <level>700</level> <!--config -->
        </target>
        <target>
          <name>com.cloudbees</name>
          <level>800</level> <!--info -->
        </target>
        <target>
          <name>com.cloudbees</name>
          <level>900</level> <!--warning -->
        </target>
        <target>
          <name>com.cloudbees</name>
          <level>1000</level> <!--severe -->
        </target>
        <target>
          <name>com.cloudbees</name>
          <level>2147483647</level> <!--off -->
        </target>
      </targets>
    </log>
EOF▼

As another example, here is how to add the needed loggers to debug SAML issues on CloudBees CI:

kubectl exec -i ${CONTROLLER_POD_NAME} -n ${CLOUDBEES_CI_NAMESPACE} -- bash -c "mkdir -p /var/jenkins_home/log && cat >/var/jenkins_home/log/kb-article.xml <<EOF
<?xml version='1.1' encoding='UTF-8'?>
<log>
  <name>kb-article</name>
  <targets>
    <target>
      <name>org.jenkinsci.plugins.saml</name>
      <level>300</level>
    </target>
    <target>
      <name>org.pac4j</name>
      <level>500</level>
    </target>
  </targets>
</log>
EOF"▼

This solution has been tested with CloudBees CI on modern cloud platforms - managed controller - 2.235.5.1

Solution 5: Configure default levels from the UI

This solution does not survive a restart

You can set the default logging level under Manage Jenkins System Log Log Levels. Simply copy/paste the logger or package you want to adjust the level for, select the logging Level and click on Submit. For example, the following matches the same configuration as in Solution 1.

Custom loggers, log levels and performance impact considerations

When changing a log level or creating a new custom logger, you should clearly understand the implications of such a change and the potential impact in the overall performance of your instance. By changing a log level from WARNING to ALL you might be changing the number of lines persisted to the logs in several orders of magnitude, and this can severely impact the performance of your instance and can even bring it to a point where it is not usable anymore.

There will be good reasons for you to increase the log levels for a given class or create a custom logger to further troubleshoot an elusive problem. Nevertheless, the general recommendation is that the change should not stay in the system, so after you enable that log level change or that custom logger, wait for some minutes (enough for the issue that you are looking for to happen an be logged) and right after get the log setting back to normal or remove the custom logger that you have created.

Sharing logs with CloudBees Support

To share the logs with CloudBees Support, generate a support bundle including the Controller Custom Log Recorders option.

In that support bundle the custom loggers will be inside the nodes/master/logs/custom folder. You can also go to your $JENKINS_HOME/logs/custom to get the same information if you have access to the instance file system.

Removing Custom Loggers

To remove a custom logger you have implemented, navigate to Manage Jenkins System Log to view your configured loggers.

Click on the cog/configure icon to adjust individual class loggers or delete the logger entirely.

References

For more information about Jenkins logging and loggers, please refer to:

https://www.jenkins.io/doc/book/system-administration/viewing-logs/