Memory problem: 'unable to create new native thread'

2 minute read

Issue

You see this stacktrace in your controller log:

Caused by: java.lang.OutOfMemoryError: unable to create new native thread
      at java.lang.Thread.start0(Native Method)
      at java.lang.Thread.start(Thread.java:714)
      at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
      at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1371)

Resolution

The controller could be encountering:

  • OS limits of

  • number of running user processes

  • number of opened files

  • A thread leak

When this issue happens, collect a thread dump as per How to generate a thread dump? and check OS limits.

To see the current limits of your system, run ulimit -a on the command-line with the user running the controller (eg cloudbees-core-cm). You should see something like this:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 30
file size               (blocks, -f) unlimited
pending signals                 (-i) 30654
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 99
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

To increase limits:

For all non-RPM package versions, and RPM versions before 2.289.2.2

  1. add these lines to /etc/security/limits.conf:

    cloudbees-core-cm      soft   nofile  8192
    cloudbees-core-cm      hard   nofile  8192
    cloudbees-core-cm      soft   nproc   30654
    cloudbees-core-cm      hard   nproc   30654

    Note that this assumes cloudbees-core-cm is the user running the controller.

  2. You can now logout and login and check that the limits are correctly modified with ulimit -a (limits are applied when the user logs in).

  3. If it’s not, check that one PAM configuration file in /etc/pam.d/ contains this line:

    session required pam_limits.so
  4. Restart the service for the changes to take effect.

For RPM package versions from 2.289.2.2 onward

  1. Run sudo systemctl edit cloudbees-core-cm

  2. Add the following entries to the unit file using your editor (refer to https://access.redhat.com/solutions/1257953), then save and exit:

    [Service]
    LimitNOFILE=8192
    LimitNPROC=30654
  3. Restart the service for the changes to take effect.