How to disable the weather column to resolve instance slowness?

Article ID:216973327
3 minute readKnowledge base

Issue

The Jenkins instance experiences an overall slowdown. Slow-request logs from the support bundle should show some stack trace involving WeatherColumn/column.jelly. It should resemble something like this: 10410msec elapsed in Handling GET /master3/ from 127.0.0.0 : RequestHandlerThread[#17] View/index.jelly WeatherColumn/column.jelly

Environment

Explanation

The weather column is a Column that can be used in Jenkins List views to displays a weather icon that represents the health of the item listed. The health is based on an aggregated score calculated mainly from build metrics.

  • In the case of a Job, the health is calculated from the job build history.

  • In the case of a Folder, the health is calculated recursively following several configurable Health Metrics.

When an instance grows to be very large, and it’s folder structure has many levels, the generation of this weather column can cost a lot of system resources slowing down other processes.

By default in Jenkins:

  • the Jenkins "All" views display all the possible column for a List View, and consequently the Weather column

  • when a Folder is created, an "All" view is created and all the possible folder health metrics are added

Therefore by default, the weather column reports the health of all items.

Resolution

If an instance is impacted by the performance of the Weather Column, a solution is to remove the folder health metrics of all existing folders.

The only caveat of such a change is that the weather column will always report Folders as Healthy.

Existing Folders

Here are some steps to disable these health metrics at the folder level:

  1. Go to all of your top-level folders.

  2. Click the configuration page.

  3. Click Health Metrics.

  4. Delete all of the entries (except Enabled Projects, it may be left if you find it useful or just want some icon in the weather column).

  5. Save the folder configuration.

The following script automates the process:

import jenkins.model.Jenkins
// try dry run first, set false to apply changes
def dryRun = true

final metricToExclude = "com.cloudbees.hudson.plugins.folder.health.ProjectEnabledHealthMetric"
final verb = dryRun ? "Would remove" : "Removing"
Jenkins.instance.allItems(com.cloudbees.hudson.plugins.folder.AbstractFolder).each { folder ->
    def removed = folder.healthMetrics.removeIf { metric ->
        if (metric.class.name.equals(metricToExclude)) {
            return false
        }
        println "${verb} ${metric.class.simpleName} from ${folder.name}"
        return !dryRun
    }
    if (removed) {
        folder.save()
    }
}
return null
▼

By default this script runs in a dry run mode, i.e., it only prints what it is going to modify. To modify the health metrics change the dryRun variable from true to false.

This will stop the weather column from constantly refreshing on all folders and the jobs within them.

Note that running this script will disable the Weather Column for existing folders only, but not for newly created folders in the future.

This can be added to a new groovy init script, such as jenkins-home/init.groovy.d/weather.groovy as well if you encounter timeouts running this via the script console.

Newly Created Folders

Folder Plugin 6.15 or later

Since the release of the Folder Plugin 6.15, folders are created without adding any health metric by default. Health metrics’s report have a very high performance impact in terms of I/O on instances with big quantity of jobs and builds inside folders. These health metrics at folder level are not usually visualized, so the community decided to disable them by default to increase the scalability of Jenkins - specially when the $JENKINS_HOME is not running on a SSD.

We can fall back to the previous behaviour in which the health metrics were added by default by injecting the Java argument -Dcom.cloudbees.hudson.plugins.folder.config.AbstractFolderConfiguration.ADD_HEALTH_METRICS=true.

Folder Plugin 6.10.1 or later

Since the release of the Folder Plugin 6.10.1, it is possible to change the folder health metrics added by default when creating a Folder.

To remove all default folder health metrics, go to Manage Jenkins  Configure System  folder and remove all the Health Metric controls like the following, then save:

global health metrics

This guarantee that all newly created folders do not have those folder health metrics so the weather column will not report their health.

Before Folder Plugin 6.10.1

There is no simple solution other than running periodically the script provided above that addressed existing folders.

The script may well be executed from Jenkins Script Console or as a scheduled job.

The script could even be used as a Cluster Operation and run across multiple connected controllers from the Operations Center. And this Cluster Operation could be run on a regular basis.

Of course, this script should be run on a test controller before running across production controllers.