Issue
-
The Jenkins logs frequently show
WARNINGS
like the followingWARNING c.c.j.s.i.c.HttpWorkersCounterPeriodicWork#doRun: Running low on resources due to few workers available
-
The
$JENKINS_HOME/support/http-workers-monitor/
directory is rapidly growing in size
Environment
-
CloudBees CI (CloudBees Core) 2.222.1.1 and later
-
CloudBees CI (CloudBees Core) on modern cloud platforms - Managed controller 2.222.1.1 and later
-
CloudBees CI (CloudBees Core) on modern cloud platforms - Operations Center 2.222.1.1 and later
-
CloudBees CI (CloudBees Core) on traditional platforms - Client controller 2.222.1.1 and later
-
CloudBees CI (CloudBees Core) on traditional platforms - Operations Center 2.222.1.1 and later
-
CloudBees Jenkins Enterprise 1.11.29 and later
-
CloudBees Jenkins Enterprise - Managed controller 2.222.1.1 and later
-
CloudBees Jenkins Enterprise - Operations center 2.222.1.1 and later
Related Issues
-
FNDJEN-3041: HttpWorkersCounterPeriodicWork is giving false positives with Jetty
-
JENKINS-60821 (cause)
Explanation
A change in the naming of the HTTP Workers threads started by Jetty Winstone (default Jetty servlet container embedded in Jenkins) causes the HTTP Workers counter probe of the CloudBees Support Plugin to report false positives shortages of HTTP workers.
This change was released in Winstone 5.7, included in Jenkins core since version 2.218 and therefore impacts CloudBees CI since version 2.222.1.1.
This probe - at the moment of writing - generates a minimal support bundle when a shortage is detected under $JENKINS_HOME/support/http-workers-monitor/
to help capture the root cause of HTTP worker thread shortage. If false positive are very frequent, several bundles will be generated and that directory will grow in size, without retention. Depending on the size / activity in those instances, this could well cause performances and disk space issues.
Resolution
The issue FNDJEN-3041 is resolved in version 3.25 of the CloudBees Support plugin that is included in CloudBees CI since version 2.249.3.1.
The solution is to:
-
upgrade CloudBees CI to version 2.249.3.3 or later
It is also recommended cleaning up the $JENKINS_HOME/support/http-workers-monitor/
directory
Workaround
As a workaround:
-
Disable the HTTP Counter probe by adding the system property
jenkins.servlet.container=disabled
on startup -
Restart Jenkins
It is also recommended cleaning up the $JENKINS_HOME/support/http-workers-monitor/
directory.
It is possible to disable the counter probe live by executing the groovy script System.setProperty("jenkins.servlet.container", "disabled") under . This does not survive a restart.
|