Data Retention

7 minute readReference

This collection of API commands is used to manage data archive activities of the enterprise. The need to archive for enterprises is primarily for run-time data such as jobs, pipeline runs, and so on because this data tends to grow more and at a must faster pace than definitions or object models such as applications. Use these commands to manage archival and purge activities for following type of run-time data:

  • Jobs

  • Releases

  • Pipeline runs

  • Deployments

Data Retention Policies

A data retention policy defines the retention criteria for a given object.

Archive Connectors

An archive connector registers the archival system with the CloudBees CD/RO archiving framework.

createDataRetentionPolicy

Creates a new DataRetentionPolicy object.

Arguments Descriptions

dataRetentionPolicyName

The data retention policy name.

Argument type: String

objectType

Type of object for which the retention rule is defined. Possible values:

  • job

  • release

  • pipelineRun

  • deployment

Argument type: Enum

age

The age of data to include in a retention operation. For example, consider a rule specifying age=45 and ageUnit=days ; previously unarchived or unpurged data older than 45 days is included in the current operation.

Argument type: Integer

ageUnit

Time unit for age. Possible values are:

  • minutes

  • hours

  • days

  • weeks

  • months

  • years

Type: Enum

action

The action to take on the data that matches the criteria for the data retention rule. Possible values are:

  • archiveOnly

  • purgeAfterArchive

  • purgeOnly

Type: Enum

description

(Optional) User-defined description for this policy. Not interpreted by CloudBees CD/RO .

Argument Type: String

enabled

(Optional) < Boolean flag0|1|true|false >

Set to true to enable this data retention policy. Defaults to true at create time.

Argument Type: Boolean

includeSubreleases

(Optional) < Boolean flag0|1|true|false >

Whether to automatically include the sub-releases for the releases matching the data retention rule. Applicable with objectType = release.

0 | false : Do not include sub-releases. (default)

1 | true : Include sub-releases.

Argument Type: Boolean

projectNames

List of possible projects to which object can belong.

Type: Collection

associatedTags

List of possible tags with which the object can be associated.

Type: Collection<String>

statuses

List of possible statuses for the object. Possible values are based on objectType and include:

  • Release: COMPLETED

  • Job, Deployment, Pipeline Run: success, warning, error, aborted

If not specified, all statuses are used.

Type: Collection

additionalFilters

Additional filter criteria for the object type similar to search filters.

Type: Collection

Positional arguments

dataRetentionPolicyName

Response

Returns a dataRetentionPolicy object.

ec-perl

syntax: $cmdr->createDataRetentionPolicy(<dataRetentionPolicyName>, {<optionals>});

Example

$cmdr->createDataRetentionPolicy('ArchivePolicy1', {objectType -> 'release', projectNames -> 'default', age -> '1', ageUnit -> 'months', action -> 'archiveOnly'});
$cmdr->createDataRetentionPolicy('ArchivePolicy1', {objectType -> 'job', projectNames -> 'default', age -> '1', ageUnit -> 'months', action -> 'archiveOnly'});

ectool

syntax: ectool createDataRetentionPolicy <dataRetentionPolicyName> [<optionals>]

Example

ectool createDataRetentionPolicy ArchivePolicy1 --objectType release --projectNames default --age 1 --ageUnit months --action archiveOnly
ectool createDataRetentionPolicy ArchivePolicy1 --objectType job --projectNames default --age 1 --ageUnit months --action archiveOnly

deleteDataRetentionPolicy

Deletes the specified dataRetentionPolicy object.

Arguments Descriptions

dataRetentionPolicyName

The name of the data retention policy to delete.

Argument type: String

Positional arguments

dataRetentionPolicyName

Response

None or a status OK message.

ec-perl

syntax: $cmdr->deleteDataRetentionPolicy(<dataRetentionPolicyName>);

Example

$cmdr->deleteDataRetentionPolicy("ArchivePolicy1");

ectool

syntax: ectool deleteDataRetentionPolicy <dataRetentionPolicyName>

Example

ectool deleteDataRetentionPolicy "ArchivePolicy1"

getDataRetentionPolicy

Retrieves the specified dataRetentionPolicy object.

Arguments Descriptions

dataRetentionPolicyName

The name of the data retention policy to retrieve.

Argument type: String

Positional arguments

dataRetentionPolicyName

Response

Returns a dataRetentionPolicy object.

ec-perl

syntax: $cmdr->getDataRetentionPolicy(<dataRetentionPolicyName>);

Example

$cmdr->getDataRetentionPolicy("Test");

ectool

syntax: ectool getDataRetentionPolicy <dataRetentionPolicyName>

Example

ectool getDataRetentionPolicy "ArchivePolicy1"

getDataRetentionPolicies

Retrieves all dataRetentionPolicy objects in the system.

Positional arguments

None

Response

Returns zero or more dataRetentionPolicy objects.

ec-perl

syntax: $cmdr->getDataRetentionPolicies();

Example

$cmdr->getDataRetentionPolicies();

ectool

syntax: ectool getDataRetentionPolicies

Example

ectool getDataRetentionPolicies

modifyDataRetentionPolicy

Modifies the specified dataRetentionPolicy object.

Arguments Descriptions

dataRetentionPolicyName

The data retention policy name.

Argument type: String

objectType

Type of object for which the retention rule is defined. Possible values:

  • job

  • release

  • pipelineRun

  • deployment

Argument type: Enum

age

The age of data to include in a retention operation. For example, consider a rule specifying age=45 and ageUnit=days ; previously unarchived or unpurged data older than 45 days is included in the current operation.

Argument type: Integer

ageUnit

Time unit for age. Possible values are:

  • minutes

  • hours

  • days

  • weeks

  • months

  • years

Type: Enum

action

The action to take on the data that matches the criteria for the data retention rule. Possible values are:

  • archiveOnly

  • purgeAfterArchive

  • purgeOnly

Type: Enum

clearAdditionalFilters

(Optional) < Boolean flag0|1|true|false >

Whether to clear all additionalFilters specified for this policy .

0 | false : Do not clear. (default)

1 | true : Clear.

Argument Type: Boolean

clearAssociatedTags

(Optional) < Boolean flag0|1|true|false >

Whether to clear all associatedTags specified for this policy .

0 | false : Do not clear. (default)

1 | true : Clear.

Argument Type: Boolean

clearProjectNames

(Optional) < Boolean flag0|1|true|false >

Whether to clear all projectNames specified for this policy .

0 | false : Do not clear. (default)

1 | true : Clear.

Argument Type: Boolean

clearStatuses

(Optional) < Boolean flag0|1|true|false >

Whether to clear all statuses specified for this policy .

0 | false : Do not clear. (default)

1 | true : Clear.

Argument Type: Boolean

description

(Optional) User-defined description for this policy. Not interpreted by CloudBees CD/RO .

Argument Type: String

enabled

(Optional) < Boolean flag0|1|true|false >

Set to true to enable this data retention policy. Defaults to true at create time.

Argument Type: Boolean

includeSubreleases

(Optional) < Boolean flag0|1|true|false >

Whether to automatically include the sub-releases for the releases matching the data retention rule. Applicable with objectType = release.

0 | false : Do not include sub-releases. (default)

1 | true : Include sub-releases.

Argument Type: Boolean

newName

(Optional) The new name for this DataRetentionPolicy.

Argument type: String

projectNames

(Optional) List of possible projects to which object can belong.

Type: Collection

associatedTags

(Optional) List of possible tags with which the object can be associated.

Type: Collection

statuses

(Optional) List of possible statuses for the object. Possible values are based on objectType and include:

  • Release: COMPLETED

  • Job, Deployment, Pipeline Run: success, warning, error, aborted

If not specified, all statuses are used.

Type: Collection

additionalFilters

Additional filter criteria for the object.

Type: Collection

Positional arguments

dataRetentionPolicyName

Response

Returns the modified dataRetentionPolicy object.

ec-perl

syntax: $cmdr->modifyDataRetentionPolicy(<dataRetentionPolicyName>, {<optionals>});

Example

$cmdr->modifyDataRetentionPolicy("Test", {newName => "NewTest"});

ectool

syntax: ectool modifyDataRetentionPolicy <dataRetentionPolicyName> [<optionals>]

Example

ectool modifyDataRetentionPolicy "Test" --newName "NewTest"

createArchiveConnector

Creates a new archiveConnector object. Note: Archive connectors re for arc.

Arguments Descriptions

archiveConnectorName

A unique name for the archive connector.

Argument type: String

archiveScript

The DSL script registered to connect to the archive system and store the data being archived. The following arguments are available to the script:

  • args.entityName : Entity being archived, for example, release, job, flowRuntime.

  • args.archiveObjectType : Object type defined in the data retention policy, for example, release, job, deployment, pipelineRun.

  • args.entityUUID : Entity UUID of the entity being archived.

  • args.serializedEntity : Serialized form of the entity data to be archived based on the configured archiveDataFormat.

  • args.archiveDataFormat : Data format for the serialized data to be archived.

The archive script must return a boolean value.

  • true —if the data was archived

  • false —if the data was not archived

Argument type: String

actualParameters

(Optional) The actual parameter values for the configured archive connector.

Argument type: String

archiveDataFormat

(Optional)The data format in which the connector consumes the archived data. Possible values are: JSON, XML.

Argument type: String

enabled

(Optional) < Boolean flag0|1|true|false >

Enabled state of the connector. Only one connector can be enabled.

0 | false : Connector is disabled. (default)

1 | true : Connector is enabled.

Argument Type: Boolean

Positional arguments

archiveConnectorName

Response

Returns the new archiveConnector object.

=== Example Archive Scripts

Out of the box, CloudBees CD/RO provides DSL for two archive connectors. Use these as starting points to customize based on your own requirements. When ready to implement your connector, save the DSL script to a file ( `MyArchiveConnector.dsl ` is used below) and run the following on the command line:

ectool evalDSL --dsl MyArchiveConnector.dsl

File Archive Connector

This connector writes data to an absolute archive directory in your file system. Use as is or customize with your own logic, for example, to store data is subdirectories by month or year.

If you customize the logic, update the example DSL and apply it to the CloudBees CD/RO server by using the following command, where fileConnector.dsl is the name of your customized DSL script:

ectool evalDsl --dslFile fileConnector.dsl

Now, enable it (in either case) with the following command:

ectool modifyArchiveConnector "File Archive Connector" --actualParameter archiveDirectory="C:/archive" --enabled true
archiveConnector 'File Archive Connector', { enabled = true archiveDataFormat = 'JSON' // Arguments available to the archive script // 1. args.entityName: Entity being archived, e.g., release, job, flowRuntime // 2. args.archiveObjectType: Object type defined in the data retention policy, // e.g., release, job, deployment, pipelineRun // 3. args.entityUUID: Entity UUID of the entity being archived // 4. args.serializedEntity: Serialized form of the entity data to be archived based on // the configured archiveDataFormat. // 5. args.archiveDataFormat: Data format for the serialized data to be archived // // The archive script must return a boolean value. // true - if the data was archived // false - if the data was not archived archiveScript = ''' def archiveDirectory = 'SET_ABSOLUTE_PATH_TO_ARCHIVE_DIRECTORY_LOCATION_HERE' def dir = new File(archiveDirectory, args.entityName) dir.mkdirs() File file = new File(dir, "${args.entityName}-${args.entityUUID}.json") // Connectors can choose to handle duplicates if they needs to. // This connector implementation will not process a record if the // corresponding file already exists. if (file.exists()) { return false } else { file << args.serializedEntity return true }''' }

DevOps Insight Archive Connector

This connector configures archiving to the CloudBees Analytics server.

If you customize the logic, update the example DSL and apply it to the CloudBees CD/RO server by using the following command, where fileConnector.dsl is the name of your customized DSL script:

ectool evalDsl --dslFile fileConnector.dsl

Enable it with the following command:

ectool modifyArchiveConnector "DevOps Insight Server Connector" --enabled true

Apply the DSL script below to create a report object type for each object that can be archived.

// Create the report objects for the archived data before creating the // archive connector for {SDA-ANALYTICS} server connector. reportObjectType 'archived-release', displayName: 'Archived Release' reportObjectType 'archived-job', displayName: 'Archived Job' reportObjectType 'archived-deployment', displayName: 'Archived Deployment' reportObjectType 'archived-pipelinerun', displayName: 'Archived Pipeline Run'

This DSL script creates the following report object types:

  • archived-release

  • archived-job

  • archived-deployment

  • archived-pipelinerun

archiveConnector 'DevOps Insight Server Connector', { // the archive connector is disabled out-of-the-box enabled = true archiveDataFormat = 'JSON' // Arguments available to the archive script // 1. args.entityName: Entity being archived, e.g., release, job, flowRuntime // 2. args.archiveObjectType: Object type defined in the data retention policy, // e.g., release, job, deployment, pipelineRun // 3. args.entityUUID: Entity UUID of the entity being archived // 4. args.serializedEntity: Serialized form of the entity data to be archived based on // the configured archiveDataFormat. // 5. args.archiveDataFormat: Data format for the serialized data to be archived // // The archive script must return a boolean value. // true - if the data was archived // false - if the data was not archived archiveScript = ''' def reportObjectName = "archived-${args.archiveObjectType.toLowerCase()}" def payload = args.serializedEntity // If de-duplication should be done, then add documentId to the payload // args.entityUUID -> documentId. This connector implementation does not // do de-duplication. Documents in DOIS may be resolved upon retrieval // based on archival date or other custom logic. sendReportingData reportObjectTypeName: reportObjectName, payload: payload return true ''' }

ec-perl

syntax: $cmdr->createArchiveConnector(<archiveConnectorName>, {<archiveScript>, <optionals>});

Example

$cmdr->createArchiveConnector('MyArchiveConnector', {archiveScript -> 'July2019Archive'});

ectool

syntax: ectool createArchiveConnector <archiveConnectorName> --archiveScript [<optionals>]

Example

ectool createArchiveConnector "MyArchiveConnector" --archiveScript 'July2019Archive'

deleteArchiveConnector

Deletes the specified archiveConnector object. You must specify the archiveConnectorName to delete.

Arguments Descriptions

archiveConnectorName

The name of the archive connector to delete

Argument type: String

Positional arguments

archiveConnectorName

Response

None or a status OK message.

ec-perl

syntax: $cmdr->deleteArchiveConnector(<archiveConnectorName>);

Example

$cmdr->deleteDataRetentionPolicy("Deploy East");

ectool

syntax: ectool deleteArchiveConnector <archiveConnectorName>

Example

ectool deleteArchiveConnector "Deploy East"

getArchiveConnector

Retrieves the specified archiveConnector object.

Arguments Descriptions

archiveConnectorName

The name of the DataRetentionPolicy category to retrieve. Use getArchiveConnectors to determine ArchiveConnectorName.

Argument type: String

Positional arguments

archiveConnectorName

Response

Returns a archiveConnector object.

ec-perl

syntax: $cmdr->getArchiveConnector(<archiveConnectorName>);

Example

$cmdr->getArchiveConnector("Default Pipelines");

ectool

syntax: ectool getArchiveConnector <archiveConnectorName>

Example

ectool getArchiveConnector "Default Pipelines"

getArchiveConnectors

Retrieves all archiveConnector objects.

Positional arguments

None

Response

Returns a list of archiveConnector objects.

ec-perl

syntax: $cmdr->getArchiveConnectors();

Example

$cmdr->getArchiveConnectors();

ectool

syntax: ectool getArchiveConnectors

Example

ectool getArchiveConnectors

modifyArchiveConnector

Modifies the specified archiveConnector object. You must specify an archiveConnectorName.

Arguments Descriptions

archiveConnectorName

A unique name for the archive connector.

Argument type: String

archiveScript

The DSL script registered to connect to the archive system and store the data being archived. The following arguments are available to the script:

  • args.entityName : Entity being archived, for example, release, job, flowRuntime.

  • args.archiveObjectType : Object type defined in the data retention policy, for example, release, job, deployment, pipelineRun.

  • args.entityUUID : Entity UUID of the entity being archived.

  • args.serializedEntity : Serialized form of the entity data to be archived based on the configured archiveDataFormat.

  • args.archiveDataFormat : Data format for the serialized data to be archived.

The archive script must return a boolean value.

  • true —if the data was archived

  • false —if the data was not archived

Argument type: String

actualParamters

(Optional) The actual parameter values for the configured archive connector.

Argument type: String

archiveDataFormat

(Optional)The data format in which the connector consumes the archived data. Possible values are: JSON, XML.

Argument type: String

clearActualParameters

(Optional) < Boolean flag0|1|true|false >

True if the archive connector should remove all actual parameters.

Argument Type: Boolean

enabled

(Optional) < Boolean flag0|1|true|false >

Enabled state of the connector. Only one connector can be enabled.

0 | false : Connector is disabled. (default)

1 | true : Connector is enabled.

Argument Type: Boolean

archiveConnectorName

A unique name for the archive connector.

Argument type: String

newName

(Optional) New name for ArchiveConnectorName.

Argument type: String

Positional arguments

archiveConnectorName

Response

Returns the modified archiveConnector object.

ec-perl

syntax: $cmdr->modifyArchiveConnector(<archiveConnectorName>, {<optionals>});

Example

$cmdr->modifyArchiveConnector("MyArchiveConnector", newName-> "MyOtherArchiveConnector");

ectool

syntax: ectool modifyArchiveConnector <archiveConnectorName> [<optionals>]

Example

ectool modifyArchiveConnector "MyArchiveConnector" --newName "MyOtherArchiveConnector"