# Data retention Data retention ============== More and more enterprises from regulated industries, like financial services and healthcare, have strict auditing guidelines around data retention. For example, some agencies need to maintain records for seven years for compliance purposes. As CloudBees CD/RO is used, the volume of runtime data accumulates in its databases, possibly eroding performance, along with data storage challenges. CloudBees CD/RO data retention provides a way for the enterprise to manage its data archival and purge needs for runtime objects. Data retention in CloudBees CD/RO includes support for these concepts: * Support for data archiving: The process of copying data to storage external to the CloudBees CD/RO server. Available via either UI or API. * Support for data purging: The process of deleting data from the CloudBees CD/RO server. Available via either UI or API. * Data retention policies: Data archive and purge criteria, based on object type. Configure via either UI or API. * Archive connectors: Specifications of the target archival systems. Configure via API, only. [](#_key_benefits_of_data_retention)Key benefits of data retention ------------------------------------------------------------------ CloudBees CD/RO data retention provides the following key benefits to your organization: * Performance and cost benefits: Archiving infrequently accessed data to a secondary storage optimizes application performance. Systematically removing or purging data that is no longer needed from the system helps improve performance and saves disk space. * Regulatory compliance: Enterprises in regulated industries, such as the Financial industry, are required to retain data for certain lengths of time for regulatory compliance. * Internal corporate policy compliance: Organizations may need to retain historical data for audit purposes or to comply with corporate data retention policies * Business intelligence and analytics: Organizations want to use archived information in potentially new and unanticipated ways. For example, in retrospective analytics, ML, and more. [](#_planning_your_data_retention_strategy)Planning your data retention strategy -------------------------------------------------------------------------------- * Decide which objects to include in your retention strategy. Supported objects include: * Releases * Pipeline runs * Deployments * Jobs * CloudBees CI builds * Workflows * Decide on data retention server settings. For more information, refer to [Setting up data retention](#_setting_up_data_retention). Keeping in mind the amount and frequency of data you wish to process, configure CloudBees CD/RO server settings to handle the rate. * Decide the archive criteria for each object type. For more information, refer to [Manage data retention policies](#manage-data-retention-policies). * List of projects to which the object can belong. * List of completed statuses for the object based on the object type. Active objects cannot be archived. * Look-back time frame of completed status. * Action: archive only, purge only, purge after archive. * Decide on the archive storage system. For more information, refer to [Managing archive connectors](#_managing_archive_connectors). The choice of archive storage systems may be driven by the enterprise’s data retention requirements. For example, regulatory compliance might require a WORM-compliant storage that prevents altering data, while data analytics might require a different kind of storage that allows for easy and flexible data retrieval. Based on your archival requirements, you can create an archive connector into which the CloudBees CD/RO data archiving process feeds data. Archive storage system types include: * Cloud-based data archiving solutions: AWS S3, AWS Glacier, Azure Archive Storage, and so on. * WORM-compliant archive storage: NetApp. * Analytics and reporting system: Elasticsearch, Splunk, and so on. * Traditional disk-based storage. * RDBMS and NoSql databases. [](#_setting_up_data_retention)Setting up data retention -------------------------------------------------------- To set up data retention for your CloudBees CD/RO server, you must perform the following: * Create your data detention rules. For more information, refer to [Manage data retention policies](#manage-data-retention-policies). * For archiving: An archive connector needs to be established. For more information, refer to [Managing archive connectors](#_managing_archive_connectors). Following is the list of CloudBees CD/RO server settings related to data retention. To access server settings, available from CloudBees CD/RO: 1. From the CloudBees navigation, select CloudBees CD/RO. 2. From the main menu, go to **Administration**  **Server Settings**  **Data Retention**. The **Edit Data Retention** page displays. Related settings include: Setting Name Description Enable Data Retention Management When enabled, the data retention management service is run periodically to archive or purge data based on the defined data retention policy. > true—archiving enabled. (default) > > false—archiving disabled. Property name: `enableDataRetentionManagement` Type: Boolean Data Retention Management service frequency in minutes Controls how often, in minutes, the data retention management service is scheduled to run. Default: `1440` (1 day) Property name: `dataRetentionManagementFrequencyInMinutes` Type: Number of minutes Data Retention Management batch size Number of objects to process as a batch for a given data retention policy in one iteration. Default: `100` Property name: `dataRetentionManagementBatchSize` Type: Number Maximum iterations in a Data Retention Management cycle Maximum number of iterations in a scheduled data archive and purge cycle. Default: `10` Property name: `maxDataRetentionManagementIterations` Type: Number Number of minutes after which archived data may be purged Number of minutes after which archived data may be purged if the data retention rule is set up to purge after archiving. Default: `10080` (7 days) Property name: `purgeArchivedDataAfterMinutes` Type: Number of minutes The `Cleanup Associated Workspaces` feature adds to the time the purge process takes to complete. The `Cleanup Associated Workspaces` feature should not be used if you are running CloudBees CD/RO versions 10.3.5 or earlier, as the feature can cause performance issues on your instance. [](#manage-data-retention-policies)Manage data retention policies ----------------------------------------------------------------- From the CloudBees CD/RO UI, browse to `https:///` and select **Administration**  **Data Management** from the main menu. Use the information detailed below to view and manage data retention policies. ![Data retention policies list](_images/data-retention-policies.566cd9d.png) Figure 1. Data retention policies 1. Manage data retention policies by: * Remove retention policies. 1. Select **policy checkbox** 2. Select **Delete data retention policy**. * Edit a data retention policy. * Select the policy name link. * Select ![Add](../_images/icons/blue-info-h-icon.2cdc388.png) policy information. 2. Review the following retention policy information: * Filter retention policies using runtime objects by selecting **Run** ![Add](../_images/icons/blk-run-icon.29d8bde.png). * Review the retention policy **Object**, **Archive**, **Purge** or **Modified** column data. 3. Configure a data retention policy by: * Creating a new data retention policy by selecting **Add a data retention policy** * Activating the policy by selecting **Enabled**. * Modify a retention policy by selecting the policy three-dots ![three-dots](../_images/icons/3-dot-icon.7cedbb7.png) and a configuration option. View modification options. * **Edit:** - Use to modify a retention policy. * **DSL Export:** - Use to download a policy in DSL format. * **Preview:** Use to view a list of objects to be archived or purged. * **Delete:** - Use to remove the retention policy. [](#_edit_a_data_retention_policy)Edit a data retention policy -------------------------------------------------------------- Access policy modification features from the **Data retention policies** page by: * Selecting the data retention policy name link or ![Add](../_images/icons/blue-info-h-icon.2cdc388.png) policy information. * Selecting **Edit** from the ![three-dots](../_images/icons/3-dot-icon.7cedbb7.png) three-dots menu. ### [](#_data_retention_policy_details)Data retention policy - Details Use the information detailed below to edit data retention policy details and definitions. ![Edit data retention policy](_images/data-retention-policy.0773aa5.png) Figure 2. Edit data retention policy - Details Modify policy details. * Type a name for the data retention policy. * Type a description for the data retention policy. ### [](#_data_retention_policy_definition)Data retention policy - Definition Define how many objects to retain. ![Edit data retention policy](_images/edit-data-retention-policy.84ffc92.png) Figure 3. Edit data retention policy - Definition - Retain object Identify the number of objects to retain. * Select **Retain object** to identify the number of objects to retain. * Enter the number of the newest, most recent objects to hold into the `Number of objects` field. For example, if the policy is to hold the 100 most recent objects, enter `100`. The result is the latest 100 objects, chronologically, are always stored while other data beyond this count are subject to the **Action** defined below. ![Edit data retention policy](_images/data-retention-definition.67396bd.png) Figure 4. Edit data retention policy - Definition - Object time frame Define the timeframe for the policy. * Select **Object time frame**. The `age` and `ageUnit` fields together define a time threshold from the current date, determining which data to preserve or act upon based on its age. When the data retention process is initiated, it uses the current date and time as its reference point ("now"). With the `age` and `ageUnit` fields, define the length of time previous to "now", instead of specifying a specific date. * Enter the time-based number that defines the timeframe for this policy into the `Number` field. * Select the time unit (Minutes, Hours, Days, Weeks, Months, or Years) from the `Age unit` field to complete the time threshold for data retention. ![Edit data retention policy](_images/define-definition-fields.795edc9.png) Figure 5. Edit data retention policy - Define remaining definition fields Select the attributes for the remaining definition fields. * Choose an object by type and status: * **Completed Releases**: Include sub-releases. * **Completed Pipeline runs**: Successful, Error, Warning, Aborted. * **Completed Deployments**: Successful, Error, Warning, Aborted, Remove deployments in database, Remove associated workspaces. * **Completed Jobs**: Successful, Error, Warning, Aborted, Remove jobs in database, Remove associated workspaces. * **Completed Builds**: Successful, Failed, Unstable, Aborted, Not built. * **Completed workflows**: Completed. * Filter by project. * Filter by tags. * Select the action to take when an object is no longer within the retention policy: **Purge**, **Archive**, **Purge after archive**. When using **Purge after archive** with **Schedule**  **Set frequency**  **Once** you may need to run the policy twice. **Purge after archive** only purges objects if the duration configured in **Number of minutes after which archived data may be purged** is met, found in **Administration**  **Server settings**  **Data retention**. The first run archives the objects, but if they have not met the duration configured in **Number of minutes after which archived data may be purged**, they are not purged immediately. To purge these archived objects, run the schedule again after the objects have been archived for the duration configured. If you want to purge objects immediately after archiving, configure **Number of minutes after which archived data may be purged** to `0`. * Indicate whether to include subreleases. ### [](#_data_retention_policy_schedule_and_more_criteria)Data retention policy - Schedule and More criteria Select the link below to use the [schedules](/docs/cloudbees-cd/latest/projects/schedules#edit-existing-schedule) feature to modify all the schedules linked to the data retention policy. ![Edit data retention policy](_images/multiple-schedules.8f505e0.png) Figure 6. Edit data retention policy - Multiple Schedules Use the information detailed below to edit the data retention policy schedule and retention object criteria. ![Edit data retention policy](_images/single-schedule.d863a7c.png) Figure 7. Edit data retention policy - Single Schedule and More criteria The save, cancel, and preview features are located at the bottom of the screen. 1. Choose one of the retention policy schedule options: * Select the **Use global schedule** checkbox tp use the default schedule. * Select the **Disable using a schedule** checkbox to turn off scheduling. 2. Define schedule attributes. * Enter a unique schedule identifier into the `Schedule name` field. * Select a project to link the schedule from the `Project` menu options. * Choose the policy run interval by selecting the `Set frequency` conditional options: * **Once** * **Every Day** * **Days of the Week** * **Days of the Month** * Complete the **Time zone**, **Start time** and **Start date** fields. * Apply the scheduling policy that defines how a schedule resumes in case the scheduled time is interrupted. * Select **Ignore the missed triggers and wait for the next schedule time** to run the job at the next scheduled time. * Select **Run once as soon as possible, then resume normal scheduling** to run the job at the soonest time that occurs within an active region. For example, at server startup a scheduled job will not trigger immediately on startup. 3. Configure the field name or root parameter object criteria to include additional field names or root parameter objects. * Select **Add criteria**. * Add the `Field name` criteria by either: * Enter the name of the field or root parameter. * Select the name from the ![blue arrow](../_images/icons/blue-arrow-menu-h-icon.75420bc.png) menu. * Select an `Operator type` option. * Enter applicable `Value` field text. * Modify existing `Field name`, `Operator type`, or `Value` field entries. * Remove criteria objects: * Select the `Field name` checkbox. * Select **Delete**. ### [](#_via_api)Via API You can create and manage data retention rules using the API commands through the `ectool` command-line interface or through a DSL script. For complete details about the API, refer to [CloudBees CD/RO APIs](/docs/cloudbees-cd/latest/api/). [](#_managing_archive_connectors)Managing archive connectors ------------------------------------------------------------ The data retention framework provides an extension mechanism to register archive connectors configured for the particular storage system to support different archival systems. CloudBees CD/RO includes two predefined sample archive connectors located in `/src/samples/data_archive/archiveConnectors.dsl`, which are disabled by default. Once a connector is enabled, use it as a starting point for developing your custom connector. The two sample connectors are: * `File Archive Connector`: Configures a directory to use as the archive target. * `CloudBees CD/RO DevOps Insight Server Connector`: Configures archiving to a report object. Only one archive connector can be enabled in the system at a time. Any data retention policies within the system that include an archive action will exclusively utilize the currently enabled connector. ### [](#_via_ui)Via UI There is no UI support for managing archive connectors. However, this page explains the command-line instructions and provides examples for using the predefined archive connectors with the API. ### [](#_via_api_2)Via API Create and manage archive connectors using the API commands through the `ectool` command-line interface or a DSL script. CloudBees CD/RO provides DSL for two archive connectors out of the box. Use one of these as a starting point to customize based on your requirements. * When you are ready to implement your connector, save the DSL script to a file ( `MyArchiveConnector.dsl` is used below) and run the following on the command line: ectool evalDSL --dsl MyArchiveConnector.dsl * To create your own archive connector, and add it to the system: ectool createArchiveConnector * To view your available archive connector: ectool getArchiveConnector #### [](#_file_archive_connector)File archive connector This connector writes data to an absolute archive directory in your file system; use the connector out of hte box, or customize its logic, for example, to store data in subdirectories by month or year. If you customize the logic, update the example DSL and apply it to the CloudBees CD/RO server by using the following command, where `fileConnector.dsl` is the name of your customized DSL script: ectool evalDsl --dslFile fileConnector.dsl #### [](#_file_archive_connector_use_case_example)File Archive Connector use case example In this File Archive Connector example, the jobs are archived into the `tmp/archive` folder in the file system, specified using the `archiveDirectory` parameter. The following criteria are used for the configuration: * Archive completed successful jobs from the 'Test project 1' project. * Select jobs older than three days. * Archive the jobs into the `/tmp/archive` folder in the file system. * Format all jobs in XML. * Do not use a custom schedule for this data retention policy. Because the predefined archive connectors are disabled by default, you must first configure the `File Archive Connector` with basic functionality, then enable it, as follows: * Set the job to be in XML data format. * Specify `/tmp/archive` as the target folder. * Enable the archive connector. Continue with the steps below to configure the `File Archive Connector` for this example. 1. Run the command to specify the requirements: ectool modifyArchiveConnector "File Archive Connector" \ --archiveDataFormat XML \ --enabled 1 \ --actualParameter archiveDirectory="/tmp/archive" 2. Review the response; it should be similar to: 098b6165-41d6-11ee-b425-0800275bae6d File Archive Connector XML def dir = new File(args.parameters.archiveDirectory, args.archiveObjectType) dir.mkdirs() File file = new File(dir, "${args.archiveObjectType}-${args.entityUUID}.${args.archiveDataFormat.toLowerCase()}") // Connectors can choose to handle duplicates if they need to. // This connector implementation will not process a record if the // corresponding file already exists. if (file.exists()) { return false } else { file << args.serializedEntity return true } 2023-08-23T16:56:43.325Z 1 admin 2023-08-28T08:37:16.973Z admin 098b8877-41d6-11ee-b425-0800275bae6d archiveDirectory /tmp/archive 3. Set up the data retention policy. The system may have several different data retention policies with different settings/filters. Based on the requirements, the data retention policy should specify: * Archive action. * 3 days age. * Job object type. * Filters such as project name 'Test project 1' and job status
 'successful'. * Data retention policy enabled. * Applicable default settings for the global schedule used by all data retention policies. 4. Run the `ectool` command to create the data retention policy using the requirements described above: ectool createDataRetentionPolicy "Data retention policy 1" \ --action archiveOnly \ --age 3 \ --ageUnit days \ --description "Archive all successfully completed jobs in the 'Test project 1' project older than 3 days into the /tmp/archive folder in the file system in XML format with help of predefined 'File Archive Connector' archive connector. Use global/general schedule for this data retention policy." \ --enabled 1 \ --objectType job \ --projectNames "Test project 1" \ --statuses success 5. Review the response; it should be similar to: 7bb8c0bd-4580-11ee-b2fb-0800275bae6d Data retention policy 1 archiveOnly 3 days 0 2023-08-28T08:54:22.881Z Archive all successfully completed jobs in the 'Test project 1' project older than 3 days into the /tmp/archive folder in the file system in XML format with help of predefined 'File Archive Connector' archive connector. Use global/general schedule for this data retention policy. 1 admin 2023-08-28T08:54:22.881Z job admin 7bb8c0bf-4580-11ee-b2fb-0800275bae6d Test project 1 success 6. Wait to ensure that all jobs are archived as expected. Your response should be similar to: tree /tmp/archive/ /tmp/archive/ └── job ├── job-82dea183-4581-11ee-b015-0800275bae6d.xml ├── job-83c9c464-4581-11ee-b2fb-0800275bae6d.xml ├── job-851704f2-4581-11ee-b2fb-0800275bae6d.xml ├── job-85b281f8-4581-11ee-8141-0800275bae6d.xml ├── job-86303d30-4581-11ee-b2fb-0800275bae6d.xml ├── job-88634a26-4581-11ee-8141-0800275bae6d.xml ├── job-89554b54-4581-11ee-8141-0800275bae6d.xml ├── job-8ab1ce11-4581-11ee-b015-0800275bae6d.xml ├── job-9637f082-4581-11ee-8141-0800275bae6d.xml ├── job-973e8b0f-4581-11ee-b015-0800275bae6d.xml ├── job-97e6b18e-4581-11ee-b2fb-0800275bae6d.xml ├── job-98aa9ddc-4581-11ee-b2fb-0800275bae6d.xml ├── job-99550eba-4581-11ee-b2fb-0800275bae6d.xml ├── job-9a21fbfd-4581-11ee-b015-0800275bae6d.xml ├── job-9af99740-4581-11ee-9231-0800275bae6d.xml └── job-9c042930-4581-11ee-8a83-0800275bae6d.xml 1 directory, 16 files In the `/tmp/archive` target folder above, a subfolder `job` is created for the archived object type, containing the archived jobs in the XML format. #### [](#_the_cloudbees_cdro_devops_insight_server_connector)The CloudBees CD/RO DevOps Insight Server Connector This connector configures archiving to the CloudBees CD/RO DevOps Insight Server (DOIS), also called the Analytics server. If you customize this connector’s logic, you must update the example DSL and apply it to the CloudBees CD/RO server using the following command, where `doisConnector.dsl` is the name of your customized DSL script: ectool evalDsl --dslFile doisConnector.dsl #### [](#_devops_insight_server_connector_use_case_example)DevOps Insight Server Connector use case example In this DevOps Insight Server Connector example, the following criteria are used for the configuration: * Archive jobs from the 'Test project 2' project. * Select jobs older than 15 minutes. * Archive the jobs into the Analytics server (DOIS). * The jobs must be in JSON format, as shown in the predefined DevOps Insight Server Connector archive connector. * Use a custom schedule for this data retention policy. Configure and enable the `DevOps Insight Server Connector`. 1. Create the DOIS component. The connector stores all data in the DOIS component via explicitly-created ElasticSearch indexes. Run the command to create the special ElasticSearch indexes: ectool evalDsl """ reportObjectType 'archived-release', displayName: 'Archived Release' reportObjectType 'archived-job', displayName: 'Archived Job' reportObjectType 'archived-deployment', displayName: 'Archived Deployment' reportObjectType 'archived-pipelinerun', displayName: 'Archived Pipeline Run' """ 2. Review the response; it should be similar to: 5eca97bf-458b-11ee-a626-0800275bae6d archived-pipelinerun 2023-08-28T10:12:18.064Z ef-archived-pipelinerun-*/_search?pretty 0 Archived Pipeline Run admin 2023-08-28T10:12:18.064Z admin 5ecabed1-458b-11ee-a626-0800275bae6d 0 0 3. Configure and enable the DevOps Insight Server Connector. Because the predefined archive connectors are disabled by default, you must first configure the `DevOps Insight Server Connector` with basic functionality, then enable it, as follows: Only one archive connector may be enabled in the system at a given time; all data retention policies with archive action in the system will use it. 4. Run the `ectool` command to specify JSON data format and enable the archive connector: ectool modifyArchiveConnector "DevOps Insight Server Connector" \ --archiveDataFormat JSON \ --enabled 1 5. Check the response; it should be similar to: 099832ae-41d6-11ee-b425-0800275bae6d DevOps Insight Server Connector JSON def reportObjectName = "archived-${args.archiveObjectType.toLowerCase()}" def payload = args.serializedEntity // If de-duplication should be done, then add documentId to the payload // args.entityUUID -> documentId. This connector implementation does not // do de-duplication. Documents in DOIS may be resolved upon retrieval // based on archival date or other custom logic. sendReportingData reportObjectTypeName: reportObjectName, payload: payload return true 2023-08-23T16:56:43.325Z 1 admin 2023-08-28T10:15:43.172Z admin 099832b0-41d6-11ee-b425-0800275bae6d 6. Set up the data retention policy. Be aware that the system may have a number of different data retention policies with different configurations present. Based on the requirements in the data retention policy, specify the following: * Archive action * 15 minute age options * Job object type * Filters such as project name 'Test project 1' and job status
 'successful' * Enable
 data retention policy 7. Run the `ectool` command to specify the requirements: ectool createDataRetentionPolicy "Data retention policy 2" \ --action archiveOnly \ --age 15 \ --ageUnit minutes \ --description "Archive all jobs in the 'Test project 2' project older than 15 minutes into the Analytics server (DOIS) in JSON format with help of predefined 'DevOps Insight Server Connector' archive connector. Use a custom schedule for this data retention policy." \ --enabled 1 \ --objectType job \ --projectNames "Test project 2" 8. Review the response, it should be similar to: de33f934-458c-11ee-b1f0-0800275bae6d Data retention policy 2 archiveOnly 15 minutes 0 2023-08-28T10:23:02.012Z Archive all jobs in the 'Test project 2' project older than 15 minutes into the Analytics server (DOIS) in JSON format with help of predefined 'DevOps Insight Server Connector' archive connector. Use a custom schedule for this data retention policy. 1 admin 2023-08-28T10:23:02.012Z job admin de33f936-458c-11ee-b1f0-0800275bae6d Test project 2 success aborted warning error 9. Set a custom schedule; this is necessary because of the expected high frequency (try to archive existing jobs at least once per 15 minutes). The `ectool` command to create a custom schedule for this particular data retention policy in the Default project: ectool createSchedule Default 'Data retention policy 2 schedule' \ --dataRetentionPolicyName 'Data retention policy 2' \ --interval 15 \ --intervalUnits minutes 10. Review the response, it should be similar to: 3f527d62-458d-11ee-8c8f-0800275bae6d Data retention policy 2 schedule 2023-08-28T10:25:44.947Z Data retention policy 2 15 minutes admin ignore 2023-08-28T10:25:44.947Z admin normal Default 3f52a474-458d-11ee-8c8f-0800275bae6d 0 Default Europe/Berlin 1 0 11. Check to ensure that jobs are archived as expected. The command below checks to see if jobs are being archived as expected. It only works if you have installed CloudBees Analytics without security. If CloudBees Analytics is secure on your system, check the server logs, or log into the CloudBees Analytics user interface as `reportuser`, and go to the following URL to view the data: `https://localhost:9201/ef-archived-job*/_search`. Also, the curl command below specifies the server host’s name (ubuntu1); replace this value in the command with the name of your server host. 12. Run the command to check that the jobs are being archived: IPv6 addresses are only supported for Kubernetes platforms. If using an IPv6 address, enclose the address in square brackets. Example: `[]`. curl http://ubuntu1:9201/_cat/indices?v 13. Review the response, it should be similar to: health status index uuid pri rep docs.count docs.deleted store.size pri.store.size yellow open ef-archived-job-2023 ilgvht5NR1-WTivVNhChUA 2 1 5 0 42.2kb 42.2kb yellow open ef-job-2023 0bIvsUeqSMWkR04pC2Zqdg 2 1 5 0 50.9kb 50.9kb As you can see in the OpenSearch, at least two indexes are created: `ef-job-2023` for regular jobs and `ef-archived-job-2023` for archived jobs. If no indexes are displayed, there is a general communication issue between CloudBees CD/RO and CloudBees Analytics. To troubleshoot: 1. Check your CloudBees CD/RO and CloudBees Analytics logs for any warnings or errors. 2. Check your CloudBees CD/RO license. 3. Try sending report data manually with the help of the API: ectool sendReportingData --help Usage: sendReportingData [--validate <0|1|true|false>] For complete details about the API, refer to [CloudBees CD/RO APIs](/docs/cloudbees-cd/latest/api/).