Memory problem: 'unable to create new native thread'

Issue

You see this stacktrace in your controller log:

Caused by: java.lang.OutOfMemoryError: unable to create new native thread
      at java.lang.Thread.start0(Native Method)
      at java.lang.Thread.start(Thread.java:714)
      at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
      at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1371)

Environment

Resolution

The controller could be encountering:

OS limits of
number of running user processes
number of opened files
A thread leak

When this issue happens, collect a thread dump as per How to generate a thread dump? and check OS limits.

To see the current limits of your system, run ulimit -a on the command-line with the user running the controller (eg cloudbees-core-cm). You should see something like this:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 30
file size               (blocks, -f) unlimited
pending signals                 (-i) 30654
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 99
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

To increase limits:

For all non-RPM package versions, and RPM versions before 2.289.2.2

add these lines to /etc/security/limits.conf:

cloudbees-core-cm      soft   nofile  8192
cloudbees-core-cm      hard   nofile  8192
cloudbees-core-cm      soft   nproc   30654
cloudbees-core-cm      hard   nproc   30654

Note that this assumes cloudbees-core-cm is the user running the controller.

You can now logout and login and check that the limits are correctly modified with ulimit -a (limits are applied when the user logs in).
If it’s not, check that one PAM configuration file in /etc/pam.d/ contains this line:
```
session required pam_limits.so
```
Restart the service for the changes to take effect.

For RPM package versions from 2.289.2.2 onward

Run sudo systemctl edit cloudbees-core-cm
Add the following entries to the unit file using your editor (refer to https://access.redhat.com/solutions/1257953), then save and exit:
```
[Service]
LimitNOFILE=8192
LimitNPROC=30654
```
Restart the service for the changes to take effect.

This article is part of our Knowledge Base and is provided for guidance-based purposes only. The solutions or workarounds described here are not officially supported by CloudBees and may not be applicable in all environments. Use at your own discretion, and test changes in a safe environment before applying them to production systems.