DSL Git Synchronization

YAML formatted DSL files is a Preview feature.

A Preview feature:

Has not undergone end-to-end testing with CloudBees products.
Is provided without service-level agreements (SLA), and therefore does not include CloudBees' commitment to functionality or performance.
May impact other stable areas of the product when used.
May have limited documentation.
May not be feature-complete during the Preview period.
May graduate from Preview to a supported feature or be removed from the product.
May introduce breaking changes that prevent upgrades due to incompatibility with future development.

Product features and documentation are frequently updated. If you find an issue or have a suggestion, please contact CloudBees Support.

As of CloudBees CD/RO v2024.03.0, DSL Git Synchronization is installed as part of the standard CloudBees CD/RO Tools. For installation instructions, refer to the CloudBees Tools installation.

The DSL Git Synchronization CLI utility extends the functionality of the generateDsl and evalDsl APIs to help manage and promote DSL across multiple files and directories in a GitOps and IDE friendly manner. This feature applies intelligence to the DSL evaluation based upon Git logs, delete or modify actions and apply only changes.

DSL Git does not support monorepositories. As a result, the following limitations are observed when executing the dslsync apply command:

Sequential processing: Projects can be applied one at a time. Execution time depends on the number of projects in the repository and the size of each project.
Non-transactional execution: The dslsync apply operation is not transactional, and any error during the operation might require you to review, fix, and synchronize again and again. For example, when applying to a complex object (a project), if one of its siblings fails (a pipeline), then the user must analyze the failure and re-run the DSLSync operation on the object (project). This challenge intensifies in mono-repo environments, where interdependencies and a large number of root objects significantly increase the surface area for potential synchronization interruptions or errors.

Additionally, unlike EC-DslDeploy, which requires an agent to run, the dslsync commands are run directly from a CLI without requiring a CloudBees CD/RO agent. While both file formats are similar, the two are not compatible. For more information about EC-DslDeploy, refer to DslDeploy plugin.

The CLI also supports YAML formatted DSL files in addition to the existing Groovy formatted DSL files. However, if no format is specified, the CLI defaults to Groovy formatted DSL files. Format specified examples include:

Groovy format: --format=groovy
YAML format: --format=yaml
Default format (Groovy): --format or no parameter included.

Repositories currently storing configuration data in Groovy DSL should not be migrated to YAML within the same repository. This migration path is unsupported and may lead to unpredictable results. To adopt YAML, CloudBees recommends starting a new repository for the YAML files.

The latest version of dslSync supports exporting in either YAML or Groovy format. However, when migrating between Groovy DSL and YAML DSL, the DSL files are renamed. In Git, this renaming is tracked as file deletions, which may appear as though all objects have been removed. If the --commit-hash-property (used to track changes) is not initialized, this migration can result in the deletion of the entire top-level object being synchronized.

To avoid this issue:

Update the repository with the YAML code (if transitioning to this format from Groovy).
Clear the existing value of the --commit-hash-property, before running dslSync apply.
Re-run with --commit-hash-property to update it correctly.

DSL Git Synchronization configurations

Use the information in these sections to configure the DSL Git Synchronization CLI.

Enable bash-completion for dslsync.
Generate the DSL Git synchronization configuration file.

Enable `bash-completion` for `dslsync`

Use the generate-completion command parameter to enable Bash-completion.

Generate the bash-completion script file by running the following command. Replace <dslsync_completion_script_dir> with the directory where the bash completion script will be generated.
```
dslsync generate-completion > <dslsync_completion_script_dir>/dslsync_completion.sh▼
```
Copy the bash-completion script.
- If bash-completion is installed, copy the bash-completion script file into one of the following directories:
  - /etc/bash_completion.d
  - /usr/local/etc/bash_completion.d
- If bash-completion is not installed.
  1. Create directory bash_completion.d in your home directory.
    
    mkdir ~/bash_completion.d
    ▼
  2. Copy the bash-completion script file into the new directory.
  3. Enter the command below into the ~/.bashrc or ~/.zshrc file.
    
    for bcfile in ~/bash_completion.d/* ; do . $bcfile done
    ▼
Start using bash-completion in one of the following ways.
- Source the ~/.bash_profile or ~/.zshrc file.
- Launch a new terminal.

Generate the DSL Git synchronization configuration file

Use the information in this section to generate the dslsync.yml configuration file. This file enables DSL code synchronization between the CloudBees CD/RO servers and Git repositories.

Check out the Git repository <initial-source> branch where the CloudBees CD/RO DSL files will be generated.
- If the <initial-source> branch exists. e.g. dev
  cd <git-repo-directory> git checkout <initial-source>
  ▼
- If the <initial-source> branch does not exist.
  cd <git-repo-directory> git branch <initial-source> git push -u origin <initial-source> git checkout <initial-source>
  ▼

Generate DSL for the following:

Generate DSL for a server object from the CloudBees CD/RO initial-source instance into the dsls sub-directory of the Git repository. For more information, refer to Server properties.

dslsync generate \
--target-directory=dsls \
 --server=<cdro-initial-source-instance>
 --object-type=server \
 --object-name="properties" \
 -  -overwrite-object-directory
 --user=<admin> --password

Generate DSL for a project object from the CloudBees CD/RO initial-source instance into the dsls sub-directory of the Git repository.

   dslsync generate \
   --target-directory=dsls \
   --server=<cdro-initial-source-instance> \
   --object-type=project \
   --object-name="<project-name>" \
-  -overwrite-object-directory
   --user=<admin> --password

Commit and push the generated DSL to the Git repository.

git add dsls
git commit -m "DSLs from initial source instance"
git push▼

Run the git log command to get the latest commit SHA.

git log -1 --stat # to get <latest-commit-SHA-for-repo-branch>▼

Locate and copy the commit SHA for use in the next step.

Run the command to create a configuration file in the directory specified by --path with the SHA content file path specified by --commit-hash-property.

If --path=dsls and --commit-hash-property=/server/dslsync/<file-name>, the following file and property will be created:

System-generated configuration file: <git-repo-directory>/dsls/.cloudbees/dslsync.yml

SHA content: /server/dslsync/<property-name>

dslsync apply \
 --path=dsls \
 --server=<cdro-initial-source-instance> \
 --commit-hash=<latest-commit-SHA-for-repo-branch> \
 --commit-hash-property=/server/dslsync/<property-name>
 --user=<admin> --password▼

dslsync apply \
 --path=dsls \
 --server=<cdro-initial-source-instance> \
 --commit-hash=<latest-commit-SHA-for-repo-branch> \
 --format=yaml \
 --commit-hash-property=/server/dslsync/<file-name>
 --user=<admin> --password▼

Commit the dslsync.yml file into the Git repository <initial-source> branch.

git add dsls/.cloudbees/dslsync.yml
git commit -m "dslsync config with tracker"
git push▼

Promote DSL configuration file from <initial-source> branch to Git branches and CloudBees CD/RO instances (for example, production or QA).

Repeat this step for all environment branches and CloudBees CD/RO instances to be used as target or source of dslsync commands.
```
cd <git-repo-directory>
git checkout <initial-source>
git pull

git checkout <target-branch>
git merge <initial-source>
git push
dslsync apply --path=dsls --server=<cdro-target-branch-instance>
--user=<admin> --password▼
```

DSL Git Synchronization commands

The DSL Git synchronization includes the following two commands: * Dslsync generate * DSLsync apply

The following topics provide detailed information about each command and its parameter descriptions.

DSLsync generate

Use the dslsync generate command to generate DSL scripts for CloudBees CD/RO objects.

Table 1. generate command
Command parameters	Description
`-- h — help`	Show this message and exit.
`-- format`	Specify YAML or Groovy formatted DSL files. If `-- format` is not used, the CLI defaults to Groovy formatted DSL files.
`-- object-name`	The name of the CloudBees CD/RO object to be DSL generated.
`-- object-type`	The type of CloudBees CD/RO object.
`-- overwrite-object-directory`	Delete the parent directory of the object before generating the DSL. If this option is not used and the parent directory is not empty, the DSL will return an error.
`-- password`	The authentication password for CloudBees CD/RO.
`-- port`	Port number to use when connecting to the agent. The default is 8443.
`-- server`	The domain name or IP address of the CloudBees CD/RO server. Enclose the IPv6 address in square brackets. Example: `[<IPv6-ADDRESS>]`
`-`- target-directory`	The directory where the generated DSL files will be created in a nested file structure.
`-- timeout`	The maximum duration of the command execution in seconds.
`-- token`	The authentication access token for CloudBees CD/RO.
`-- user`	The authentication username for CloudBees CD/RO.
`-- v — verbose`	Include verbose output.

DSLsync apply

Use the dslsync apply command to apply DSL scripts from a Git repository to a CloudBees CD/RO server.

Table 2. apply command
Command parameters	Description
`--disable-git-rename-detection`	Prevents false positives during synchronization in repositories with frequent or large-scale refactoring.
`-- dryrun`	Prints the evaluated DSL without sending it to the server.
`-- force`	Applies changes to the CloudBees CD/RO server without prompting or without using the contents of the `dslsync_pre.dsl` file.
`-- format`	Specifies DSL files in YAML or Groovy format. If `-- format` is not provided, the CLI defaults the DSL files to Groovy format.
`-- generate-prerun-dsl`	Generates `dslsync_pre.dsl` file containing changes from the current working directory. These changes are not applied to the server.
`-- h — help`	Displays help message and exits.
`-- password`	The authentication password for CloudBees CD/RO.
`-- path`	The path to the Git repository directory containing the scripts. The default is the current directory.
`-- resume-from`	Applies the DSL starting from the specified script. All scripts that run before this are skipped. Use this option if a previous run stopped due to an unexpected error.
`-- timeout`	Specifies the maximum execution time in seconds.
	`-- token`
The authentication access token for CloudBees CD/RO.	`-- user`
The authentication username for CloudBees CD/RO.	`-- v — verbose`

List of compatible DSL API objects

The following DSL API objects are valid values for --object-type option. For more information about each API object, refer to CloudBees CD/RO DSL Objects

Table 3. DSL API objects
DSL API Objects
archiveConnector
artifact
ciConfiguration
complianceConfiguration
dataRetentionPolicy
directoryProvider
emailConfig
gateway
group
openIDConnectConfiguration
persona
personaCategory
personaPage
project
reportObjectType
repository
resource
resourcePool
samlIdentityProvider
samlServiceProvider
serviceAccount
tag
user
workspace
zone

DSL API implementation for YAML import or export

DSLsync CLI provides the following implementation support for API objects. Ensure the following are implemented in the API configuration:

Add valid authentication token for complianceConfiguration API
Provide public certificate and private key for samlServiceProvider API
Manually enable directoryProvider API

Add valid authentication token for complianceConfiguration API

To import and export complianceConfiguration object in YAML format, update the configuration file with a valid authentication token either through the user interface or manually. Applying only the YAML file does not activate compliance.

Provide public certificate and private key for samlServiceProvider API

To import and export samlServiceProvider object in YAML format, provide a public certificate encoded in Base64 format and the corresponding private key to ensure proper SAML configuration and validation. The YAML file must be populated with these values before applying the resource.

Manually enable directoryProvider API

To import and export directoryProvider object in YAML format, you must manually enable it in the user interface after applying the YAML.

Usage examples

Use the following examples to:

Add DSL from CloudBees CD/RO instance to a Git repository
Apply DSL from a Git repository to CloudBees CD/RO instance
Promote DSL changes from Git to CloudBees CD/RO instances and a Git repository

Known limitation

When using YAML DSL with dslsync, Git change detection might differ from Groovy DSL during bulk operations such as mass renames, deletions, or additions across multiple YAML files.

Due to structural differences between YAML and Groovy representations, Git might interpret large-scale YAML changes as complete file deletions and recreations rather than incremental modifications. As a result, when dslsync compares the previous commit SHA with the the current commit, it might detect a larger or different set of changes than expected.

This behavior does not impact the correctness of the synchronization but it might cause dslsync to perform more operations than expected.

This limitation applies only to bulk YAML file updates (for example, large refactors, directory restructures, or multi-file renames). Standard incremental YAML file edits are not affected.

Add DSL from CloudBees CD/RO instance to a Git repository

Use the dslsync generate command to create DSL for a top-level CloudBees CD/RO object in a Git branch.

Check out the Git repository <target> branch where the CloudBees CD/RO DSL files will be generated.
- If the <target-branch> exists (for example, dev).
  cd <git-repo-directory> git checkout <target-branch>
  ▼
- If the <target> branch does not exist.
  cd <git-repo-directory> git branch <target-branch> git push -u origin <target-branch> git checkout <target-branch>
  ▼

Generate DSL for a CloudBees CD/RO object into the dsls sub-directory of the Git repository.

dslsync generate \
 --target-directory=dsls \
 --server=<cdro-source-instance> \
 --object-type=<top-level-object-type> \
 --object-name="<object-name>" \
 --user=<admin> --password▼

dslsync generate \
 --target-directory=dsls \
 --server=<cdro-source-instance> \
 --object-type=<top-level-object-type> \
 --object-name="<object-name>" \
 --format=yaml \
 --user=<admin> --password▼

Apply DSL from a Git repository to CloudBees CD/RO instance

Perform the following steps to apply committed DSL file changes from the Git repository branch to a CloudBees CD/RO instance.

Check out the Git repository <source> branch containing the CloudBees CD/RO DSL file changes.
```
cd <git-repo-directory>
git checkout <source>
git pull▼
```

Apply DSL to the CloudBees CD/RO instance.

dslsync apply --path=dsls \
--server=<cdro-target-branch-instance>▼

dslsync apply --path=dsls \
--server=<cdro-target-branch-instance>
--format=yaml▼

Promote DSL changes from Git to CloudBees CD/RO instances and a Git repository

Check out the Git repository <source> branch containing the CloudBees CD/RO DSL to be promoted.

cd <git-repo-directory>
git checkout <source>
git pull

git checkout <target-branch>
git merge <source>
git push
dslsync apply --path=dsls --server=<cdro-target-branch-instance>▼

Apply DSL to the target CloudBees CD/RO instance.

dslsync apply --path=dsls \
--server=<cdro-target-branch-instance>▼

dslsync apply --path=dsls \
--server=<cdro-target-branch-instance> \
--format=yaml▼