KBEC-00038 - Improving CloudBees CD (CloudBees Flow) server performance by tuning memory allocation

Description

CloudBees CD (CloudBees Flow) is distributed to work with a minimum memory configuration of 2 GB. Most modern server configurations have the ability to add more memory, which CloudBees CD can be tuned to use. Also, you can tune CloudBees CD to use less memory to prevent disk thrashing that is associated with excess paging on a 1 GB memory configuration. You can tune your configuration’s memory to improve CloudBees CD server performance.

Java (JRE) solution

The CloudBees CD server is a Java application. As the CloudBees CD load increases, the small, default Java heap size is exhausted, which results in the following commander.log file message:

When the CloudBees CD server starts, a configuration file specifies the default Java heap size. On 32-bit systems, heap size can be set to a maximum of 1500 MB. If you have a large memory machine, increase maxmemory to the 1500 MB maximum. On small memory machines, specify a memory allocation as low as 256 MB to prevent thrashing.

Here is a list of JVM memory considerations:

Out of the box, the CloudBees CDJVM heap size is configured to use 25% of the physical memory on the machine.
In addition to the heap size, there’s memory used by code, stacks, threads, etc., so the grand total usage may exceed 25%.
Once the JVM has allocated memory from the OS, it never gives it back.
CloudBees CD will configure it to allocate the 25% right away. Growing the heap can be expensive, and this is a server product, so that makes sense as a default.
The JVM will tend to accumulate garbage until it nears the heap size limit, at which time it will garbage collect.
Because of this behavior, it often looks like the JVM is using more memory than it actually is, especially if you’re looking at the virtual size of the process, as opposed to the RSS/working set.
On an 8GB machine, a 2.x GB is expected, based on the 25% number above.

After changing the configuration file, restart the CloudBees CD server. On Linux, use the script "commander restart" in /etc/init.d. On Windows, use the Control Panel, Services tab to restart the CloudBees CD server.

For version 8.5.0 and above, the installer will recognize what type of install is taking place and will apply the following settings:

For the standalone agent install the memory values will be:

init: 16MB
max: 64MB

If any other components (server, web server, repository) is also installed, then memory values will be:

init: 256MB
max: 512MB

Java configuration example

The Java heap size attribute is located in the wrapper.conf file within the CloudBees CD server/conf directory.

As distributed, the Java heap size is a percentage of memory.

Notes:

You can configure the % or MB via ecconfigure and we should persist these values across an upgrade.
```
 ./ecconfigure --serverMaxMemory 30
 ./ecconfigure --serverMaxMemoryMB 1000
```
The wrapper memory values are a percentage of memory instead of a fixed memory value. This facilitates an easy general purpose install. On enterprise production CloudBees CD installs, it is best to place a physical number for the Java memory allocation.

On Linux use the -Xmx command to determine the maximum memory setting of Java.

 java -Xmx2500m # gives usage page (which is OK)
 java -Xmx3000m # produces error message "The specified size exceeds the maximum representable size"

The number of threads allocated in the server varies based on the number of cores in the system and how thread stacks consume address space.

Here is the extract of text from the file wrapper.conf where the changes are to be made:

# Initial Java Heap Size (in MB)
wrapper.java.initmemory=512

# Maximum Java Heap Size (in MB)
# If this is set too high, java will not run at all and the
# Commander server will not start. At least for 32-bit Windows,
# the limit is about 1500 MB.
wrapper.java.maxmemory=512

32-bit systems notes

There are memory limitations considerations and it can be expanded to a maximum of 1500 MB on Windows and perhaps more on large memory 32-bit Linux machines or decreased to 256 MB for small memory machines

Systems with many cores (say 8 or more), lower the maximum heap size on the 32-bit JVM by around 100 MB to prevent the JVM from running out of address space when there really is enough memory available. On Windows, this means lower it to 1400 MB and on Linux lower it to 2300 MB.