Jetty LowResourceMonitor triggered causing WebSocket connections to be dropped

Agent connections, or controller connections to an operations center using WebSocket transport are being dropped unexpectedly, and you see either of the following logs:

When the Jetty thread group is low on resources, there is a monitor that will cause the WebSocket timeout to become 1 second, which can cause WebSocket transport connections to be dropped (agent connections, or controller connections to an operations center).

There could be multiple reasons for this issue, so when this happens, please collect Required Data: CloudBees CI hang or high CPU on Kubernetes or Linux, and contact CloudBees Support.

One example scenario is as follows:

If you see Low Resources: Server low on threads: 200, idleThreads:0, it indicates that too many agents are being (re)connected to the controller at almost the same time. This scenario has been reproduced by CloudBees connecting 300+ WebSocket agents to the controller simultaneously. Instead of connecting all agents simultaneously but connected gradually over a period of time (say 2min), the issue does not occur, as Jetty would always have enough threads to accept the new connections. Jetty will reset the idle timeout back to its default of 30 seconds after the message INFO o.e.j.server.LowResourceMonitor#monitor: Low Resources cleared appears (typically in about 15 to 20 seconds). If you see this error frequently, it suggests there are underlying problems causing too many agents to disconnect and reconnect at once, which should be investigated and resolved. If in your specific environment, it is expected for a high number of agents to connect in a short period, you can try adjusting the Jenkins command-line argument --jettyAcceptorsCount=1 or --jettyAcceptorsCount=2 (etc.) by trial and error. Note: Do not set --jettyAcceptorsCount to high values. It is recommended to contact CloudBees Support for assistance with this configuration.

CloudBees Support will review the thread dumps captured to understand what plugin or setting could be causing the low number of threads in the Jetty thread pool.