Memory problem: 'unable to create new native thread'

Issue

You see this stacktrace in your controller log:

Caused by: java.lang.OutOfMemoryError: unable to create new native thread
      at java.lang.Thread.start0(Native Method)
      at java.lang.Thread.start(Thread.java:714)
      at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
      at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1371)

Environment

Resolution

The controller could be encountering:

OS limits of
number of running user processes
number of opened files
A thread leak

When this issue happens, collect a thread dump as per How to generate a thread dump? and check OS limits.

To see the current limits of your system, run ulimit -a on the command-line with the user running the controller (eg cloudbees-core-cm). You should see something like this:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 30
file size               (blocks, -f) unlimited
pending signals                 (-i) 30654
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 99
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

To increase limits:

For all non-RPM package versions, and RPM versions before 2.289.2.2

add these lines to /etc/security/limits.conf:

cloudbees-core-cm      soft   nofile  8192
cloudbees-core-cm      hard   nofile  8192
cloudbees-core-cm      soft   nproc   30654
cloudbees-core-cm      hard   nproc   30654

Note that this assumes cloudbees-core-cm is the user running the controller.

You can now logout and login and check that the limits are correctly modified with ulimit -a (limits are applied when the user logs in).
If it’s not, check that one PAM configuration file in /etc/pam.d/ contains this line:
```
session required pam_limits.so
```
Restart the service for the changes to take effect.

For RPM package versions from 2.289.2.2 onward

Run sudo systemctl edit cloudbees-core-cm
Add the following entries to the unit file using your editor (refer to https://access.redhat.com/solutions/1257953), then save and exit:
```
[Service]
LimitNOFILE=8192
LimitNPROC=30654
```
Restart the service for the changes to take effect.