Http Workers Counter Probe reports false-positives Warnings with Jetty Winstone

Article ID:360048187372
2 minute readKnowledge base

Issue

  • The Jenkins logs frequently show WARNINGS like the following

    WARNING c.c.j.s.i.c.HttpWorkersCounterPeriodicWork#doRun: Running low on resources due to few workers available
  • The $JENKINS_HOME/support/http-workers-monitor/ directory is rapidly growing in size

Environment

  • FNDJEN-3041: HttpWorkersCounterPeriodicWork is giving false positives with Jetty

  • JENKINS-60821 (cause)

Explanation

A change in the naming of the HTTP Workers threads started by Jetty Winstone (default Jetty servlet container embedded in Jenkins) causes the HTTP Workers counter probe of the CloudBees Support Plugin to report false positives shortages of HTTP workers.

This change was released in Winstone 5.7, included in Jenkins core since version 2.218 and therefore impacts CloudBees CI since version 2.222.1.1.

This probe - at the moment of writing - generates a minimal support bundle when a shortage is detected under $JENKINS_HOME/support/http-workers-monitor/ to help capture the root cause of HTTP worker thread shortage. If false positive are very frequent, several bundles will be generated and that directory will grow in size, without retention. Depending on the size / activity in those instances, this could well cause performances and disk space issues.

Resolution

The issue FNDJEN-3041 is resolved in version 3.25 of the CloudBees Support plugin that is included in CloudBees CI since version 2.249.3.1.

The solution is to:

  • upgrade CloudBees CI to version 2.249.3.3 or later

It is also recommended cleaning up the $JENKINS_HOME/support/http-workers-monitor/ directory

Workaround

As a workaround:

  • Disable the HTTP Counter probe by adding the system property jenkins.servlet.container=disabled on startup

  • Restart Jenkins

It is also recommended cleaning up the $JENKINS_HOME/support/http-workers-monitor/ directory.

It is possible to disable the counter probe live by executing the groovy script System.setProperty("jenkins.servlet.container", "disabled") under Manage Jenkins  Script Console. This does not survive a restart.