Issue
Agent connections to a controller or controller connections to an operations center using WebSocket transport are being dropped unexpectedly, and you see either of the following logs:
WARNING o.e.j.server.LowResourceMonitor#monitor: Low Resources: Check if the ThreadPool from monitored connectors are lowOnThreads and if all connections number is higher than the allowed maxConnection
WARNING o.e.j.server.LowResourceMonitor#monitor: Low Resources: Server low on threads: 200, idleThreads:0
Resolution
When the Jetty thread group is low on resources, there is a monitor that will cause the WebSocket timeout to become 1 second, which can cause WebSocket transport connections to be dropped (agent connections, or controller connections to an operations center).
There could be multiple reasons for this issue, so when this happens, please collect Required Data: CloudBees CI hang or high CPU on Kubernetes or Linux, and contact CloudBees Support.
One example scenario is as follows:
If you see Low Resources: Server low on threads: 200, idleThreads:0, it indicates that too many agents are being (re)connected to the controller at almost the same time.
This scenario has been reproduced by CloudBees connecting 300+ WebSocket agents to the controller simultaneously.
Instead of connecting all agents simultaneously but connected gradually over a period of time (say 2min), the issue does not occur, as Jetty would always have enough threads to accept the new connections.
Jetty will reset the idle timeout back to its default of 30 seconds after the message INFO o.e.j.server.LowResourceMonitor#monitor: Low Resources cleared appears.
If you see this error frequently, it suggests there are underlying problems causing too many agents to disconnect and reconnect at once, which should be investigated and resolved.
If in your specific environment, it is expected for a high number of concurrent agents, you can try adjusting the Jenkins command-line argument to increase the number of default threads above 200. Set --qtpMaxThreadsCount=400. Increase the qtpMaxThreadsCount gradually to cope with the number of concurrent agents connecting. The thread count does not need to match the number of expected agents. Avoid setting this value unneccessarily high. Also ensure the resource sizing of the controller is suitable.
It is recommended to contact CloudBees Support for assistance with this configuration. CloudBees Support will review the thread dumps captured to understand what plugin or setting could be causing the low number of threads in the Jetty thread pool.