Summary
Determining the correct memory setting for your CloudBees CD (CloudBees Flow) instance can take some trial-and-error.
the commander.log contains memory information which can help give you perspective on the memory usage of your system, to allow you to better configure these settings.
Description
The Java heap size attribute is located in the wrapper.conf file within the server/conf directory.
By default, the settings are set as follows:
|
For example, if your server were running on a machine with 8GB of memory, there would be approximately 3.2GB of heap memory available for the CloudBees CD (CloudBees Flow) server.
The commander.log will contain a snapshot memory capture every 15 minutes. If you search for the phrase "MemoryGuard" in the logs, you can run into such lines. This table exposes an example of what is being reported inside the commander.log file:
Pool Type Used Committed Init Max Code Cache usage 37.03 MB 37.44 MB 2.44 MB 48.00 MB Code Cache peak 37.09 MB 37.44 MB 2.44 MB 48.00 MB G1 Eden Space usage 1.516 GB 2.340 GB 2.520 GB -1 B G1 Eden Space collection 2.234 GB 2.355 GB 2.520 GB -1 B G1 Eden Space peak 8.898 GB 9.387 GB 2.520 GB -1 B G1 Survivor Space usage 184.00 MB 184.00 MB 0 B -1 B G1 Survivor Space collection 168.00 MB 168.00 MB 0 B -1 B G1 Survivor Space peak 368.00 MB 368.00 MB 0 B -1 B G1 Old Gen usage 7.182 GB 9.480 GB 9.480 GB 12.000 GB G1 Old Gen collection 5.198 GB 6.641 GB 9.480 GB 12.000 GB G1 Old Gen peak 8.282 GB 9.480 GB 9.480 GB 12.000 GB G1 Perm Gen usage 108.41 MB 112.00 MB 20.00 MB 128.00 MB G1 Perm Gen collection 104.63 MB 108.00 MB 20.00 MB 128.00 MB G1 Perm Gen peak 128.00 MB 128.00 MB 20.00 MB 128.00 MB Collector Count Time G1 Young Generation 4780 528743 G1 Old Generation 10 11698
From this table we can confirm the following information:
-
Use the Old Gen lines to learn that:
-
The currently committed memory is 9.48 GB [ see usage line ]
-
The max possible memory is 12 GB [ see MAX column in any of the 3 lines ]
-
The current memory in use is 7.182 GB [ see usage line ]
-
The maximum memory used since the last restart is 8.282 GB [ see peak line ]
-
The last time there was a Garbage Collection (GC) event, memory usage shrank back to 5.198 GB [ see collection line ]
-
-
The collector lines expose:
-
That there have been just under 5000 collection events of Young Generation memory
-
This is memory that is intended to be transient, and therefore collected frequently.
-
By dividing the time by the count, you can find that such events are taking an average of 110ms.
-
-
That there have been 10 collections of Old Generation memory. This memory is a traditional JAVA GC event where the Java engine cleans up stale memory. Under such events, the CloudBees CD (CloudBees Flow) server will temporarily stop being functional - so keeping this time to a minimum is important. By dividing the time by the count, this example shows that such events were taking 1.2s. As such, these GC events are probably not a noticeable event in most cases.
-
Should any of this example data cause you to be concerned?
If the current data is a valid representation of what’s been running in your environment for awhile, then this example implies that one might be able to lower their max memory needs to 9GB or 10GB without impacting performance on the server. The result of having a lower max memory may cause JAVA to collect old-gen memory slightly more frequently, which can help keep the time needed for GC events to a minimum.
What if I am seeing peak memory events where my USED memory reaches close to my maximum?
If this situation is a rare event, you should look into what may have caused these temporary memory peak to occur. Certain actions may cause you to encounter temporary memory peaks. eg:
-
Exports of large projects
-
Exports of a full DB w/o jobs (exports with jobs are no longer guaranteed to succeed on a running system)
-
Launching large amounts of concurrent jobs or workflows
If I am continuing to see peak events happen, should I just increase the memory I allocate to CloudBees CD (CloudBees Flow)?
Adding more memory may help to survive such peaks, but depending on your situation, and what you are trying to achieve, you may also want to consider splitting your work across multiple servers (High Availability feature is available as of v5.0). In general, if you move to HA, 3 servers should be recommended so that a node loss can be survived by the remaining 2 servers.
What are some common physical memory allocations?
A small production system may be okay with using 12-16GB of physical memory. A common allocation of memory for CloudBees CD (CloudBees Flow) customers is 32-48GB of physical memory. We have some customers who have allocated >100GB of physical memory to handle the peak-hour loads that their system encounters.