Issue
-
Your instance is running out of space due to a lot of Support Bundles being kept under the JENKINS_HOME/support folder.
-
You need to check how often the Support Bundle are being auto-generated or need to confirm the retention algorithm.
Environment
-
CloudBees Core on modern cloud platforms - Managed controller
-
CloudBees Core on modern cloud platforms - Operations Center
-
CloudBees Jenkins Enterprise
-
CloudBees Jenkins Enterprise - Managed controller
-
CloudBees Jenkins Enterprise - Operations center
Resolution
Support Bundles can be generated on demand from the Jenkins UI. These Bundles will help us troubleshoot potential issues on your instances.
Apart from the manually generated Bundle, Jenkins will auto-generate Bundles on a regular basis. These auto-generated Bundles will be stored under the $JENKINS_HOME/support/
folder. To check what is the current frequency these Bundles are being auto-generated, you can run the below script from the Script Console:
println(com.cloudbees.jenkins.support.SupportPlugin.AUTO_BUNDLE_PERIOD_HOURS)
You can also change this value so it is generated with a different periodicity. To do so, you will need to add the below startup parameter to your instance, indicating the value in hours:
-Dcom.cloudbees.jenkins.support.SupportPlugin.AUTO_BUNDLE_PERIOD_HOURS=1
Please refer to this How to add Java arguments to Jenkins article if you need further details about how to add the above startup parameter.
Please note that the Support Bundle auto-generation can be disabled by setting the "AUTO_BUNDLE_PERIOD_HOURS" to 0 in the above argument, but we strongly recommend not disabling it, as it will allow you to provide us with a Bundle even when your instance is down or the UI is unresponsive, by simply collecting one of those from the $JENKINS_HOME/support/
and attaching it to the support ticket. Having this information is key to the troubleshooting process. If you believe the periodic automatic generation of support Bundles is impacting your instances' performance, rather than completely disabling it, we would recommend you first to reduce its creation frecuency to 12 or 24 hours for example.
If the generation is working as expected but you spot any issues in the number of Bundles being kept in the $JENKINS_HOME/support/
folder, this may be related to the way Jenkins discards old Bundles. The algorithm that rotates the Bundles doesn’t keep Bundles in a linear way. Instead it follows an exponential decay/ageing algorithm. This means that it will keep several Bundles from the same hour, a Bundle from the last 2 hours, from the last 2/4 hours, a Bundle from 4/8 hours ago, a Bundle from 8/16 hours ago, 16/32, 32/64, 64/128, and following this exponential decay.
Additionally, Jenkins will generate a new Bundle on every restart, so having your instance restarted several times in a short period of time will also increase the number of Bundles stored in $JENKINS_HOME/support
. So instead of changing the Bundle auto-generation frequency, we would encourage you to confirm if your instance has been restarted several times recently, and correlate those restarts with the Bundles creation dates, that can be checked in the support Bundle file names.
Having this in mind, you should be careful when disabling or reducing the generation frequency, to avoid not having Support Bundles that would help us troubleshoot potential issues in the future. Instead, you should confirm what is the current generation frequency, remove the older ones, or the ones that are no longer required, and then monitor the Bundle generation while you confirm if new restarts are being performed in your instance.
If you are running Support core plugin versions < 2.70 and you see logs similar to the one below. You are hitting a known bug with the Support core plugin that prevents it from removing older bundles as expected with the exponential decay algorithm.
Could not save support bundle Also: java.io.IOException: This archive contains unclosed entries. at org.apache.commons.compress.archivers.zip.ZipArchiveOutputStream.finish(ZipArchiveOutputStream.java:467) at org.apache.commons.compress.archivers.zip.ZipArchiveOutputStream.close(ZipArchiveOutputStream.java:966) at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:389) org.apache.commons.compress.archivers.zip.Zip64RequiredException: <file> size exceeds the limit of 4GByte. at org.apache.commons.compress.archivers.zip.ZipArchiveOutputStream.checkIfNeedsZip64(ZipArchiveOutputStream.java:653) at org.apache.commons.compress.archivers.zip.ZipArchiveOutputStream.handleSizesAndCrc(ZipArchiveOutputStream.java:641) at org.apache.commons.compress.archivers.zip.ZipArchiveOutputStream.closeArchiveEntry(ZipArchiveOutputStream.java:514) at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:366) at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:298) at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:294) at com.cloudbees.jenkins.support.SupportPlugin$PeriodicWorkImpl.lambda$doRun$0(SupportPlugin.java:886) at java.lang.Thread.run(Thread.java:748)
Workaround
As at the time of writing a fix is yet to be released. If you observe the above stacktrace while generating a bundle in your instance the workaound is to limit the size of jenkins logs to less than 4GB following this KB article Log file rotation.