Issue
You see this stacktrace in jenkins.log:
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1371)
Environment
-
CloudBees CI (CloudBees Core) on traditional platforms - Client controller (CORE-CM)
-
CloudBees CI (CloudBees Core) on traditional platforms - Operations Center (CORE-OC)
-
CloudBees Jenkins Team (CJT)
-
Linux, Solaris
Resolution
Great chances are you are hitting one of the following system OS limit:
-
number of running user processes
-
number of opened files
To see the current limits of your system, run ulimit -a on the command-line with the user running Jenkins (usually jenkins). You should see something like this:
core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 30 file size (blocks, -f) unlimited pending signals (-i) 30654 max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 99 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 1024 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
To increase limits, add these lines to /etc/security/limits.conf:
jenkins soft nofile 4096 jenkins hard nofile 8192 jenkins soft nproc 30654 jenkins hard nproc 30654
Note that this assumes jenkins is the Unix user running the Jenkins process.
You can now logout and login and check that the limits are correctly modified with ulimit -a (limits are applied when the Unix user logs in).
If it’s not, check that one PAM configuration file in /etc/pam.d/ contains this line:
session required pam_limits.so