Diagnostic/Treatment
The following recommendation and techniques help to improve and troubleshoot performance issues on startup. They are based on experience with CloudBees customers.
Preconditions
Before troubleshooting any further, we recommend going through the following recommendations that address common causes of slowness on startup.
Apply JVM Recommendation
The speed of Jenkins startup depends on the amount of initial memory allocated. It is important that you follow the recommendation:
-
Ensure that the recommended settings are applied to the process running Jenkins, see Prepare CloudBees CI for Support.
-
Ensure that an appropriate initial heap size of memory is allocated directly on startup (i.e.
-Xms). If this is not specified, the process starts with a low portion of the available physical memory (that can vary depending on OS distribution and properties as explained in Garbage Collector Ergonomics)
Quiet Down mode on startup
Any build that were in the queue when Jenkins was stopped as well as any resumable builds (i.e. Pipeline) will be restarted on startup. This can contribute to the startup time and load, especially in environments where nodes are provisioned on demand. To avoid this:
-
Set the controller in Quiet Down mode on startup as explained in How can I prevent jenkins from starting new jobs after a restart?
| You may cancel the quiet down mode anytime when the controller has started. |
Remove Custom Loggers
Loggers are a good troubleshooting tool but they can impact the performances of your instance if they do not target a specific component or if they are specified at a high level package like for example hudson.model or hudson.security.
-
If you have created custom loggers, remove them.
Review Hook scripts
Check whether there is any post-initialization scripts under $JENKINS_HOME/init.groovy.d/ that could impact the startup of your instance. See Groovy Hook Scripts.
| Hook scripts time are actually captured by the startup log mentioned above. |
Adjust jenkins.InitReactorRunner.concurrency
During start of Jenkins, loading of jobs in parallel have a fixed number of threads by default (twice the CPU). To make Jenkins load time decrease you can use tune the System Property jenkins.InitReactorRunner.concurrency.
RunIdMigrator occurrences
If seeing jenkins.model.RunIdMigrator#migrate logs on start up, please have a look at How to Troubleshoot the Root Cause of RunIdMigrator occurrences.
Data Collection
In order to troubleshoot the startup performance further:
-
Activate the startup performance logging by adding the system property
-Djenkins.model.Jenkins.logStartupPerformance=trueon startup -
Restart Jenkins
While Jenkins is starting:
-
Collect performance data as explained in Required Data: Jenkins Hang Issue On Linux that should contain output of the script
collectPerformanceData.sh
Once Jenkins has started:
-
Collect a Support Bundle
-
Collect the Jenkins logs since the beginning of the startup
-
Collect the GC logs
If Jenkins does not start and hang:
-
Collect the latest auto-generated support bundle (see Generating a support bundle when Jenkins will not start)
-
Collect the Jenkins logs since the beginning of the startup
-
Collect the GC logs
| You may have a look at How to analyze Jenkins Startup Performance logs to understand how to analyse the startup logs |
Considerations
Please also consider the following scenario in which the Jenkins startup can be impacted.
Upgrades
After an upgrade, it is not unusual to experience slower startup caused by plugins upgrades and migration processes. Look at the logs to understand if Jenkins is doing something and what it is doing.
Restore Operations
Restore operations will show a different behavior depending on whether you are using modern platforms: CloudBees CI (CloudBees Core) or traditional platforms: CloudBees CI (CloudBees Core) on traditional platforms - Client controller and * CloudBees CI (CloudBees Core) on traditional platforms - Operations Center.
While in traditional platforms, the restore process will only impact startup slowness and will not generally show any other symptoms, in modern platforms it can cause some additional errors that will potentially cause the instance to fail provisioning due to different failures in the liveness / readiness probes.
You can get additional details on how to solve these issues in Why is my restored controller not coming up?.
Cleanup Large Instances
The startup time is proportional to the size of your instance (number of items, nodes, installed plugins). Check our recommendations for Controller Sizing.
Cleanup Old Builds
Ensure you have a strategy for cleanup as explained in the article Deleting Old Builds - Best Strategy for Cleanup and disk space management
Obsolete plugins
An instance with a large amount of plugins may take a lot of time to load all the components. It is a good practice to review plugin usage and remove obsolete ones. The CloudBees Plugin Usage can help to identify plugin usages.
Storage File System
Jenkins configuration files are stored on disk. Startup performances - and also overall performances for that matter - can be severely impacted if the storage File System holding $JENKINS_HOME is not tuned properly. If the $JENKINS_HOME does not resides in a local File System, ensure that the File System mount is tuned for performances. For NFS, please have a look at the recommendation in the article NFS Guide