Issue
Poor performance of a Jenkins instance is frequently due to misconfiguration during installation and are often the result of not following Best Practices.
Solution
Jenkins Best Practices is a great entry point and contains a collection of tips, advice, gotchas and advice for getting the most from your Jenkins instance, including performance.
The following are some good best practices to follow on your Jenkins journey.
Prepare CloudBees CI for Support
Ensure CloudBees CI follows to minimum requirements to function correctly described under Prepare CloudBees CI for Support.
Subscribe your controller to Jenkins Health Advisor by CloudBees
Jenkins Health Advisor by CloudBees identifies numerous issues before they affect your users, including security vulnerabilities, performance problems, and plugin version issues. Best of all, Jenkins Health Advisor by CloudBees is constantly being improved by the CloudBees Support and Engineering teams to cover the most recent identified issues.
Review your plugins
CloudBees suggests that you review the plugin usage and remove any plugins you are not using. See How to determine if a plugin is in use
Built-In Node (formerly known as "master" node) build executors
Never use the controller to build jobs, as this puts an unnecessary strain on resources. Disable building on the controller by navigating to Manage Jenkins -> Manage Nodes -> Built-In Node (formerly known as "master" node) and do the following:
-
Set-up # of executors to 0
-
Change the Usage strategy to Only build jobs with label expressions matching this node
Provisioning limits for your Nodes
The resources available to the Jenkins instance may enforce a practical upper limit on the number of nodes available to the instance at any given moment in time. Selecting this option permits you to set an upper limit, above which cloud providers are prevented from provisioning additional resources. As per the help text in the product, there are two important notes for this setting:
-
Limits are enforced only after the limit has been reached. For example, if the limit is set to 5 nodes, and a request to provision 6 nodes is made, then the nodes will be provisioned successfully but all further provisioning requests will be denied..
-
Limits are based upon the number of successfully connected nodes. For example, if the node limit is set to 1 and it takes 5 minutes for a node to wind up connected to Jenkins, any other provisioning request during that 5-minute window will be accepted, which may result in more than 1 node being connected.
This setting is available from Manage Jenkins -> Configure System -> Provisioning limits
Use Webhooks to trigger your builds
When possible, use Webhooks as explained in SCM Best Practices > Triggers: Polling must die section.
If for some reason, you cannot get away from SCM polling then you should limit the concurrent SCM polling to no more than 10 under Manage Jenkins -> Configure System [SCM Polling -> Max # of concurrent polling]
The problem is usually that Jenkins users create bad SCM pollings like * * * * * [poll every minute].
The below script can be executed under Manage Jenkins->Script Console to provide the SCM polling value of all the jobs configured in the instance.
import hudson.triggers.*;
import hudson.maven.MavenModuleSet;
println("--- SCM Polling for FreeStyle jobs ---");
List<FreeStyleProject> freeStyleProjectList = Jenkins.getInstance().getAllItems(FreeStyleProject.class);
for (FreeStyleProject freeStyleProject : freeStyleProjectList) {
SCMTrigger scmTrigger = freeStyleProject.getSCMTrigger();
if (scmTrigger!= null) {
String spec = scmTrigger.getSpec();
if (spec != null) {
println(freeStyleProject.getFullName() + " with spec " + spec);
}
}
}
println("--- SCM Polling for Maven jobs ---");
List<MavenModuleSet> mavenModuleSetList = Jenkins.getInstance().getAllItems(MavenModuleSet.class);
for (MavenModuleSet mavenModuleSet : mavenModuleSetList) {
SCMTrigger scmTrigger = mavenModuleSet.getTrigger(SCMTrigger.class);
if (scmTrigger!= null) {
String spec = scmTrigger.getSpec();
if (spec != null) {
println(mavenModuleSet.getFullName() + " with spec " + spec);
}
}
}
Use GitHub App authentication for GitHub Integrations
There are several reasons for using GitHub App authentication for GitHub Integration including:
-
The rate limit for a GitHub App scales with your organization size, whereas a user based token has a limit of 5000 requests per hour, regardless of how many repositories you have.
-
For organizations that have 2-factor authentication enforced - with GitHub Apps there is no need to manage 2 factor authentication tokens for a 'bot' user.
-
Jenkins GitHub Apps require a minimal, controlled set of privileges compared to a service user and its personal access token.
JobConfigHistory is a performance killer
The JobConfigHistory Plugin is one of the most used plugins, and often it is not well configured, which can produce performance issues. Thus, configure this plugin as explained in JobConfigHistory Plugin Best Practices.
For Auditing Jenkins configuration changes at Jenkins, it is more recommended to delegate this activity to your Git server by using CloudBees Configuration as Code.
Caching method for folder health
Since version 5.14 of the CloudBees+Folders+Plugin a better caching method for folder health was implemented, however we still recommend to disable the weather column for increased performance.
Post Build: Archive the Artifacts outside the JENKINS_HOME directory
Do not use the Archive the Artifacts post build step for large artifacts (> 100KB), they should be sent to your favorite Artifact Repository Manager e.g. S3, Artifactory, Nexus, etc.) and should not be kept in the Build directory of the job (e.g ${ITEM_ROOTDIR}/builds/<build>/archive)
Discard Old Builds/Fingerprints
Builds and fingerprints are correlated. Jenkins maintains a database of fingerprints, Jenkins records which builds of which projects used. This database is updated every time a build runs and files are fingerprinted. Therefore it is a best practice to manage old builds to prevent your $JENKINS_HOME directory from becoming unnecessarily large.
REST API
It is very common to see instances which are receiving a large amount of REST API calls without the Jenkins Administrators knowledge.
The number of REST API calls received can be monitored by utilizing the CloudBees Monitoring Plugin and can be accessed via Web Browser at: http/s://<INSTANCE_URL>/monitoring.
The Jetty or Tomcat access logs should also be reviewed to monitor REST API activity.
To block all the REST API calls you can use the CloudBees Request Filter Plugin and follow the instructions found here: Block All API Calls.
If your business is relying on REST API, then you should follow the best practices Best Practice For Using Jenkins REST API
Install CloudBees Disk Usage Simple to monitor JENKINS_HOME disk usage
Install the CloudBees Disk Usage Simple to get metrics regarding Disk Usage on the top of the Jenkins Metrics which comes installed by default.
Are you Suffering From Performance Issues?
Please file a support ticket on the CloudBees Support Portal and attach the required information outlined in Required Data: High CPU On Linux
If your archive is larger than 20Mb please use this service to send it to us. This service works best in Chrome or Firefox.