Log file rotation

Article ID:221764048
3 minute readKnowledge base

Issue

  • The log files in /var/log/jenkins on my Jenkins controller are rapidly growing and using up disk space. What is the best way to roll over logs without impacting the Jenkins operations and version upgrades?

  • I need to configure log rotation on my Windows Jenkins Instance.

  • I need to prevent my Garbage Collection Logs from filling up my disk space.

Environment

Resolution

Linux Hosts

The log file location may vary depending on your system. For most Linux systems, by default the Jenkins logs should be in /var/log/jenkins

The logrotate utility manages how the log files are rotated and compressed. Type man logrotate at the command-line to see a detailed list of the various options, including some good examples. Or alternatively, this Understanding logrotate Utility article offers a pretty good summary and includes examples.

The logrotate utility includes some frequently used options, such as compression, timeframe, and size of a log before rotation occurs.

If you are not encountering issues with log rotation and sizes, and for most small to medium-sized Jenkins instances - it’s probably best to let logrotate work with it’s default behavior. Here is an example of a configuration which includes good basic settings:

/var/log/jenkins/jenkins.log {
        weekly
        copytruncate
        missingok
        rotate 52
        compress
        delaycompress
        notifempty
}

Normally this file would be found in */etc/logrotate.d/jenkins. If the file does not already exist, create it and populate it with the above example.

If you have a larger Jenkins instance, or if the logs are located on a NAS (perhaps linked to /var/log/jenkins), beware that latency or connection issues between the Jenkins controller and the NAS/filesystem may result in logging issues.

Also, please ensure the Jenkins Administrator is the owner and has the correct read/write permissions to the directory and filesystem.

Windows hosts

Windows does not offer a native solution for log rotation. However, there are still options available that will allow you to achieve log rotation on Windows.

PowerShell Script

If you are experienced with writing PowerShell scripts, you can use PowerShell to create a logger and run this as a scheduled task within Windows.

Here are some resources that you may find useful for configuring logging through PowerShell.

Third-Party Applications

If you prefer to use an application, there are a few different applications that the Windows community recommends.

  • LogRotateWin is a Windows port of the logrotate utility available for Linux.

  • Logwot8 is a logrotate implementation for Windows. It is packaged with logrotate, Cygwin, and other related tools. This is a standalone log rotation solution.

Generic

Garbage Collection Logs

GC logs are enabled per our Best Practices and with the current implementation of Java 8, logs are not deleted. Therefore, it is the duty of the Linux Administrator to implement logrotate to ensure that GC logs are removed.

For troubleshooting purposes, 2-3 days of GC logs is enough to diagnose and troubleshoot an issue in most cases.

Container Logs

Note that this generally does not apply to Containers. Our docker images are built such that logs are streamed to the standard output. It is the responsibility of the container runtime to handle the logs. See Logging Architecture on Kubernetes for an example.

Related Articles: