Performance Consequences of Change Tracking

4 minute read

Change Tracking affects how CloudBees Flow runs. The impact can be from low (minimal changes to the times to update the database) to high (significantly longer times to update the database).

CloudBees Flow needs to store additional information in the database about the tracked objects in the Change History. In addition to storing normal information about CloudBees Flow operations, CloudBees Flow stores a separate archival copy of every state of every tracked object as well as the indexing information used to retrieve the Change History of tracked objects and the objects they own. This increase in database usage has performance consequences.

For most of CloudBees Flow operations, the performance consequences of Change Tracking are insignificant, typically a few percent slower in one of these situations:

  • None of the affected objects are tracked (job records and similar runtime-only objects are not tracked).

  • Only a small number of tracked objects are changed by operations.

However, for operations that affect a very large number of tracked objects in a single operation (such as importing, cloning, or deleting a large project for which change tracking is enabled), the performance consequences will be significant. The performance consequences are similar when enabling or disabling Change Tracking for a large project, because both operations require the server to write to the database the records for every tracked object owned by the project.

For these Change Tracking-intensive operations, the system performance will typically be significantly slower because several times as much data is being read and written between the CloudBees Flow server and the database. In our testing, we have seen a wide range of degrees of performance decrease from 1.25x to 10x slower when Change Tracking is enabled on a large project, compared to a project on which Change Tracking is disabled. The decrease in performance is typically from 2x to 6x.

Use Case 1: Importing and Cloning Large Projects with Change Tracking Enabled

CloudBees Flow does the following to increase the performance of Change Tracking-intensive operations in a large project with this configuration:

  • The project contains over 20,000 audited objects.

  • Change Tracking is enabled.

To increase the performance when a large project is imported or cloned, CloudBees Flow automatically decreases the amount of Change History indexing information that it saves, reducing the level of detail for these operations in the Change History. While this can improve the system performance, it can also make it harder when you want to revert an object to a specific state or to find information in the Change History while troubleshooting or debugging an issue. If you want suppress this behavior, use --reducedDetailChangeHistory false in the import or clone API command.

By default, the objects listed in the table below and the tracked objects that they own are always tracked when Change Tracking is enabled (except when Change Tracking is globally disabled).

During an upgrade from CloudBees Flow 5.0.x to CloudBees Flow 6.0, when the CloudBees Flow server starts, the server writes to the database a record for every tracked object and the objects that it owns. The objects listed in the table get new audit records. If any of the objects owns many tracked objects (such as properties), the startup process will be slow.

Objects (Entities) Path

server

/server

system objects

/systemObjects/*

users

/users/*

groups

/group/*

plugin

/plugins/*

email configuration

/emailConfigs/*

workspaces

/workspaces/*

zones

/zones/*

gateways

/gateways/*

agents

/agents/*

resources

/resources/*

resource pools

/resourcesPools/*

repositories

/repositories/*

directories

/directories/*

Use Case 2: Change Tracking of Non-Project-Owned Objects

In this context, non-project-owned objects include all objects not contained within a project. Such objects include resources, artifacts, users, groups, zones, and other objects as well as the property sheets, properties, access control lists (ACLs), and ACL entries (ACEs) for them. In releases earlier tha CloudBees Flow 6.2, Change Tracking was enabled by default for non-project-owned objects, and there was no way to disable Change Tracking for them. When users upgraded from earlier versions of CloudBees Flow to CloudBees Flow 6.0.x, the upgrade process initialized and created Change Tracking-related data for them in the database. The performance impact was usually low, except when there was a very large number (in the several hundred thousands) of non-project-owned objects, such as many properties under the /server directory; in this situation, the performance of the first startup was significantly slower. The best solution to this situation is to relocate most of the projects under a project and disable Change Tracking for that project, which may not be a practical short-term solution.

Starting in CloudBees Flow 6.2, Change Tracking can be disabled for all non-project-owned objects. (Change Tracking for these objects is enabled by default.) Using this feature is not recommended as a long-term solution; instead we recommend moving most of the objects under a project that has Change Tracking disabled.(Repeatedly disabling and enabling Change Tracking for non-project-owned project is expensive and time-consuming.)

In CloudBees Flow 6.2, when there are many non-project-owned objects, the best way to disable Change Tracking for all non-project-owned objects is to change the database.properties configuration file to disable Change Tracking for these objects before starting the server running a pre-CloudBees Flow 6.2 release. In the database.properties file, make sure that it has this line:

COMMANDER_DB_NON_PROJECT_AUDITING_ENABLED=false

Then install CloudBees Flow 6.2 and start the server, running the system this way until most of the non-project-owned objects are relocated to a non-tracked project, and re-enable Change Tracking for non-project-owned objects, which requires restarting the server.