Http Workers Counter Probe reports false-positives Warnings with Jetty Winstone

Issue

The Jenkins logs frequently shows WARNINGS like the following

  WARNING	c.c.j.s.i.c.HttpWorkersCounterPeriodicWork#doRun: Running low on resources due to few workers available▼

The $JENKINS_HOME/support/http-workers-monitor/ directory is rapidly growing in size

Environment

CloudBees CI (CloudBees Core) 2.222.1.1 and later
CloudBees CI (CloudBees Core) on modern cloud platforms - Managed controller 2.222.1.1 and later
CloudBees CI (CloudBees Core) on modern cloud platforms - Operations Center 2.222.1.1 and later
CloudBees CI (CloudBees Core) on traditional platforms - Client controller 2.222.1.1 and later
CloudBees CI (CloudBees Core) on traditional platforms - Operations Center 2.222.1.1 and later
CloudBees Jenkins Enterprise 1.11.29 and later
CloudBees Jenkins Enterprise - Managed controller 2.222.1.1 and later
CloudBees Jenkins Enterprise - Operations Center 2.222.1.1 and later
CloudBees Jenkins Platform - Client controller 2.222.1.1 and later
CloudBees Jenkins Platform - Operations Center 2.222.1.1 and later
CloudBees Support plugin
Jetty Winstone

Related Issues

FNDJEN-3041: HttpWorkersCounterPeriodicWork is giving false positives with Jetty
JENKINS-60821 (cause)

Explanation

A change in the naming of the HTTP Workers threads started by Jetty Winstone (default Jetty servlet container embedded in Jenkins) causes the HTTP Workers counter probe of the CloudBees Support Plugin to report false positives shortages of HTTP workers.

This change was released in Winstone 5.7, included in Jenkins core since version 2.218 and therefore impacts CloudBees CI since version 2.222.1.1.

This probe - at the moment of writing - generates a minimal support bundle when a shortage is detected under $JENKINS_HOME/support/http-workers-monitor/ to help capture the root cause of HTTP worker thread shortage. If false positive are very frequent, several bundles will be generated and that directory will grow in size, without retention. Depending on the size / activity in those instances, this could well cause performances and disk space issues.

Resolution

The issue FNDJEN-3041 is resolved in version 3.25 of the CloudBees Support plugin that is included in CloudBees CI since version 2.249.3.1.

The solution is to:

upgrade CloudBees CI to version 2.249.3.3 or later

It is also recommended cleaning up the $JENKINS_HOME/support/http-workers-monitor/ directory

Workaround

As a workaround:

Disable the HTTP Counter probe by adding the system property jenkins.servlet.container=disabled on startup
Restart Jenkins

It is also recommended cleaning up the $JENKINS_HOME/support/http-workers-monitor/ directory.

It is possible to disable the counter probe live by executing the groovy script System.setProperty("jenkins.servlet.container", "disabled") under Manage Jenkins Script Console. This does not survive a restart.