# Manage artifacts Manage artifacts ================ During the development process, project components or "outputs" are produced for consumption by other projects or the main project to create a whole entity. Applications are assembled by combining these various outputs (dependencies) and packaging them together into a deployable software application. Outputs can be libraries, scripts, graphics, and so on. This output is an artifact that can be consumed by other projects. Every stage in the application lifecycle produces and consumes artifacts. Artifacts are critical to the _build-test-deploy_ process. Using artifacts can: * Improve performance across builds. * Provide better reusability of components. * Improve cross-team collaboration with greater traceability. For example, instead of each developer repeatedly downloading third-party packages from external sources, these components can be published and versioned as an artifact. A developer then simply retrieves a specific artifact version from a local repository, which guarantees a consistent package from build to build. [](#Artifact_Management_objects)About artifact objects ------------------------------------------------------ CloudBees CD/RO has three types of objects to support Artifact management functionality: **_artifact_**, **_artifact version_**, and **_repository_**. Similar to other CloudBees CD/RO objects, each of these objects supports custom properties and access control. ### [](#_artifact)Artifact An **_artifact_** is a top-level object containing artifact versions, a name template for _published_ artifact versions, artifact specific properties, and access control entries to specify privileges. If an artifact is deleted, all published artifact versions within that artifact are also deleted. You can create artifacts in several ways: * Select the **Add artifact** button on the Artifacts page. * Use the `createArtifact` API command with the command-line tool (`ectool`) or a cb-perl script. * An artifact is created when you publish an artifact version if the artifact does not already exist and the user issuing the publish command has permission to create artifacts. For more information, refer to [Configuring access control](#Access_Control) and [Publishing artifact versions](#Publishing_Artifact_Versions). To create an artifact, you enter a Group ID and Artifact key. Together, these create an artifact name in the form `groupId:artifactKey`. You cannot modify the name after CloudBees CD/RO creates the artifact name from your supplied Group ID and Artifact Key specifications. Plan your artifact Group IDs and Artifact keys carefully to prevent the extra work of deleting a name and beginning your specifications again. The use of a `groupId` and `artifactKey` allows two namespaces for added flexibility in organizing artifacts within your company or organization. #### [](#_examples)Examples * If your development team uses GWT for web development and JUnit for unit testing, you may want to create `ThirdParty:GWT` and `ThirdParty:JUnit` artifacts. * Your organization may have two different development teams (for example, OS and Apps) who want to use artifacts. In building each product, both development groups want to publish and use an SDK with their product, but each SDK is different and unique. One solution is to create one artifact named `os:SDK` and another artifact named `apps:SDK`. By using different Group IDs, each development team is free to name their artifacts however they choose, without worrying about name collisions with artifacts produced by the other team. To reference an artifact, you can use the `groupId` and `artifactKey` combination or simply use the `artifactName`. The `/artifacts/` property path allows you to access any properties of a given artifact. ### [](#_artifact_version)Artifact version An **artifact version** is a collection of `0` to `N` files that were published to an artifact repository. Artifact version metadata is stored in an `artifactVersion` object in the CloudBees CD/RO server. Tips for working with artifact versions: * To group a collection of files (typically from build output in a CloudBees CD/RO workspace) into an artifact version, specify the "from" directory containing those files. * To include files from multiple directories in your artifact, you need to specify a common root directory to include these directories when publishing your artifact. * To restrict which files and subdirectories in the "from" directory to include in the artifact version, you can specify "include" and "exclude" patterns. * Artifact versions are stored on an artifact repository server in either uncompressed `tar` archives or compressed `tar-gzip` archives. * Artifact versions can be created using the `publishArtifactVersion` API, which includes creating the object in the CloudBees CD/RO server and publishing the artifact version to the artifact repository. Interactively creating an artifact version using the automation platform web UI is not supported. * When you retrieve an artifact version, the original directory structure of the directory where files were published "from" is preserved. * When you retrieve/publish artifacts, directories containing symlinks are always resolved; they are not preserved. Artifact versions are referenced by Group ID, Artifact key (or artifact name) and a version string. * The artifact version’s name is set based on the name template on the "owning" artifact. By default, the name template is in the form `groupId:artifactKey:version`. * Each artifact version name must be unique within a CloudBees CD/RO server installation. The property path `/artifactVersions/` allows access to any properties for a particular artifact version. Because you may not know the artifact version name (because of the name template), this path supports the form `::` as a substitute for the name of the artifact version. All CloudBees CD/RO APIs that take an `artifactVersionName` argument also accept this alternate form as a substitute, similar to how other APIs that accept a `jobId` argument accept the job name as a substitute. #### [](#Version-strings)Version strings A version string must be provided when publishing an artifact version. For example: `5.8.8-EN-55842` or `5.8.8.55842-EN`: In either form, the version string is interpreted as: Major version number: 5 Minor version number: 8 Patch level number: 8 Qualifier: EN Build number: 55842 Leading zeros are not valid for the major, minor, and patch versions. You must provide separator punctuation. When interpreting (parsing) the first version string form, CloudBees CD/RO interprets the text after the first hyphen as the build number if this string contains numeric characters only. Otherwise, the string is interpreted as the `qualifier`. Version string examples and how CloudBees CD/RO interprets them: Input (version string) Major Minor Patch Qualifier Build Number 1 `2.0.3-44834` `2` `0` `3` `` `44834` 2 `3.8.4-RC1` `3` `8` `4` `RC1` `0` 3 `3.8.4-RC1-36` `3` `8` `4` `RC1` `36` 4 `3.8.4.36` `3` `8` `4` `` `36` 5 `1-36` `1` `0` `0` `` `36` 6 `3.8.4.36-RC1` `3` `8` `4` `RC1` `36` In this table: * Row 5 shows that you are not required to specify all three of `` when publishing an artifact version if your organization’s versioning conventions do not use all three fields. However, there is a caveat with this level of flexibility. You can create two artifact versions in the system with equivalent versions. Version string "1-36" is equivalent to "1.0-36", but they are different, so you can publish two artifact versions with these version strings successfully. When retrieving an artifact version, in some cases, it could be unclear as to exactly which one you get. For this reason, CloudBees recommends that you adopt a convention and abide by it for a particular artifact. The artifact version object exposes components of the version string by using these intrinsic properties: * `version`: The version component is the combination of `major.minor.patch-qualifier-buildNumber`. A version component does not need to include all the following intrinsic properties. Refer to the table above. * `majorMinorPatch`: This is the `major.minor.patch` version component as specified in the version string. For example, if you specified a version string of "1-36", the `majorMinorPatch` component is "1", not "1.0.0", which makes it possible for you to meet your naming conventions by reconstituting the version string a different way in the artifact version name template. * `qualifier`: The qualifier component of the version. * `buildNumber`: The build number component of the version. #### [](#_the_artifactversionstate_property)The `artifactVersionState` property Artifact versions have an `artifactVersionState` property whose value can be one of the following: `publishing`: The artifact version is currently being published and is not available for retrieval. `available`: The artifact version is available for retrieval if needed. `unavailable`: The artifact version is not available for retrieval while in this state. You can manipulate the state between available and unavailable on the Edit Artifact Version screen. This action can be useful if an artifact version seems to be corrupted in some way, but you want to investigate before potentially deleting it from the system. By making the artifact version unavailable, retrievers can acquire an older, (potentially more stable) artifact version while you look into the problem. If the artifact version is good, you can make it available again. ### [](#_artifact_repository)Artifact repository The artifact repository is a machine where artifact versions are stored. The repository server is configured to store artifact versions in a directory referred to as the repository backing store. By default, the backing store is the `/repository-data` directory in the repository installation. You can change this default setting by running `ecconfigure --repositoryStorageDirectory` on the repository server. The repository server listens to port 8200 for HTTPS requests to publish and retrieve artifact versions. You cannot provide a mapped drive path when defining a backing store. For example, the path in the following command is not allowed and will cause artifact publishing to fail: ecconfigure --repositoryStorageDirectory c:/repository-data You must enter a UNC path instead. For example, the path in the following command is allowed: ecconfigure --repositoryStorageDirectory //10.0.109.72/repo_share/repository-data Connection information is stored in the repository object on CloudBees CD/RO. You can create repository objects in several ways: * Select the **Add repository** button in the CloudBees CD/RO UI. * Use `createRepository` API. * Repository objects can be created automatically during the CloudBees CD/RO installation. * When you are installing a repository _server_, the installer creates a corresponding repository _object_ on your CloudBees CD/RO server. * If you are installing a repository _server_ on the same machine as your CloudBees CD/RO server, a repository _object_ named "default" is automatically created. * If you are installing a repository _server_ on a different machine than your CloudBees CD/RO server, the installer prompts you for information it uses to create the corresponding repository _object_. When installing a repository server, you should install an agent also to ease maintenance tasks such as clearing out stale artifact versions from the repository backing store. 1. Navigate to **DevOps essentials**  **Artifact management**  **Repositories** to view the list of repositories. * Re-order the repository search order (for retrieving artifact versions), by selecting the six-dot icon and dragging the icon in the left column to the row position you prefer, effectively re-ordering the repository list. * For distributed environments, where the preferred repository search order varies depending on which resource is performing the retrieval, you can specify a preference order on the Edit Resource panel. For more information, refer to [https://docs.cloudbees.com/docs/cloudbees-cd/latest/resources/#create-new-edit-resource](https://docs.cloudbees.com/docs/cloudbees-cd/latest/resources/#create-new-edit-resource). The **Gateway(s):** field that appears in the **Resource Details** panel is not editable (not even for a gateway resource) and therefore does not appear in the **Edit Resource** panel. This field is automatically completed and is only for gateway agents. For example, in the **Resource Details** panel for agentp1-gw-01 and agentp2-gw-01 resources as they form a gateway Default\_to\_zone1\_gateway using the steps in the [KBEC-00393 - Setting up an environment with two zones and a gateway](/d/kb-360032824792) KB article. 2. Edit the resource by doing one of the following: * Select the name of the repository to edit from the list. * Select the **Details** link from the **Actions** menu of the repository to edit. 3. Update the fields as necessary. 4. Select **OK** when done. In the following example, a resource in California (A1) uses the "default" repository search order (that is, the order on the **Repositories** page). A1 searches for artifact versions by first requesting an artifact version from Repository 1. If the artifact version requested is not found, it then requests the artifact version from Repository 2. The resource in New York (A2) has a specific search order specified where it searches for artifacts from Repository 2, first, then searches Repository 1. ![search for repositories](_images/search-for-repositories.4031480.jpg) #### [](#_using_multiple_repositories)Using multiple repositories CloudBees CD/RO supports multiple repositories, however you cannot share a repository between multiple CloudBees CD/RO servers. Multiple repository servers may be required in your organization because of business or organizational reasons, government and compliance requirements, or for performance and data protection. For example: * Business or organizational structure: Within a company, two different development teams want to have artifact repositories containing artifacts and artifact versions specific to their product. A repository sever is created for each team. The two teams publish artifact version to their respective repositories. * Performance: A company has multiple development sites. Instead of having a single repository server, which creates significant network performance issues from retrieving artifact versions during the build process, they create a repository at each development location and have artifact versions synchronized between the repository servers. In this way, when artifact versions are retrieved for use by a build process, they are retrieved from a local repository. * Data protection and disaster recovery: A company has a single CloudBees CD/RO server with two artifact repository server machines. The contents of the repository servers are synchronized and replicated between both machines. In the event repository 1 is unavailable during a retrieve, repository 2 is searched for the requested artifact retrieval. #### [](#_copying_artifacts_to_multiple_repositories)Copying artifacts to multiple repositories You can synchronize artifacts from a source artifact repository to one or more target repositories. This helps to support distributed architectures with multiple CloudBees CD/RO artifact repository servers. You must specify the exact artifact and its version as input. This feature uses the EC-Artifact plugin, which is bundled with CloudBees CD/RO. To synchronize artifacts to other repositories, do the following: 1. Navigate to **DevOps essentials**  **Plugin management**  **Installed plugins**. 2. Select the desired artifact from the list, in this case, EC Artifact. 3. Select the **Run** icon to run the SyncArtifactVersion procedure. ![Artifact tab](_images/sync-artifact-version.a722f34.png) Figure 1. Run Procedure / SyncArtifactVersion ##### [](#_enter_run_parameters)Enter run parameters 1. Enter the following parameters as needed. Parameter Description Artifact Artifact name. The format is `:`. Version Full artifact version string. The format is `..--`. For example, `5.8.8-EN-55842`. The version string must be unique across all of this artifact’s versions. For more information, refer to the [Version strings](#Version-strings) section. Source Repository Name of the repository from which this artifact version is retrieved. Target Repository Name of the repository where the artifact is copied. You can add more repositories by selecting **Add target repository**. Overwrite? (Optional) Specifies whether to overwrite any existing artifact with the same name and version on the target repositories. This option is disabled by default. Upload in parallel? (Optional) Specifies whether to copy artifacts to multiple target repositories in parallel (for faster performance) instead of serially. You should choose whether to use this option based on how the number of target repositories and the artifact size could impact network traffic. This option is disabled by default. Priority (Optional) Priority of the job. The choices are **low**, **normal**, **high**, or **highest**. Impersonation (Optional) Credential impersonation. The choices are: * **Use pre-defined credential:** Uses the predefined credential. This is the default. * **Use specific credential:** You must specify a project name and a credential name. * **Use a specific user:** You must specify a username under which you want to log in and the password for the username to be used for LDAP testing. 2. Select **Run**. ##### [](#_understanding_overwrite_behavior)Understanding overwrite behavior The following table illustrates the overwrite behavior. Repository Result with Overwrite disabled Result with Overwrite enabled Repository 1 (source): Repository containing artifact1 to be copied – – Repository 2 (target): Repository without artifact1 artifact1 is copied to Repo2 artifact1 is copied to Repository 2 Repository 3 (target): Repository containing artifact1 artifact1 on Repository 3 is not overwritten artifact1 on Repository 3 is overwritten Repository 4 (target): Repository that does not exist Error Error ##### [](#_troubleshooting)Troubleshooting The SyncArtifactVersion procedure finishes with an error in the following situations: * Repository is unreachable: The target repository might be down. * Insufficient access rights: Access rights for writing to the target repository might be missing. #### [](#Configure-artifact-repository-server-Amazon-S3)Configuring the artifact repository server for Amazon S3 You can configure the repository server to use Amazon S3 as the backing store and connect to Amazon S3 using IAM roles. This section describes the changes to make in the `/opt/cloudbees/sda/conf/repository/server.properties` file. Before starting this procedure, make sure the following have been created and are available on the AWS Management Console ([https://aws.amazon.com/console/](https://aws.amazon.com/console/)): * An Amazon S3 bucket for the backing store. * The IAM role with a policy that grants all the `s3:\*` access to the Amazon S3 bucket you want to use with the repository server. For example, the IAM role called `s3repositoryFullAccess` has a policy called `s3fullaccessAndAccessToInstallers` that grants all the `s3:\*` access to the _backing store bucket_ called `ec-test-repository-backingstore`. Set up the artifact repository for Amazon S3, by doing the following: 1. In the `/opt/cloudbees/sda/conf/repository/server.properties` file, set `REPOSITORY_BACKING_STORE` to use Amazon S3 by changing REPOSITORY_BACKING_STORE=repository-data to REPOSITORY_BACKING_STORE=s3:// 1. Get the `` by navigating to the **IAM**  **Roles** `` page on the AWS Management Console. 2. Select **Show Policy** to open the **Show Policy** window to view the name of the backing store bucket. In the following example, `` is `ec-test-repository-backingstore:` ![s3amazon show policy](_images/s3amazon-show-policy.b463a90.png) 2. Restart the artifact repository server by running: sudo systemctl restart commanderRepository 3. To test the `REPOSITORY_BACKING_STORE`, create two files in a `/tmp/` directory, publish the directory, and verify publication using commands like this: cd /tmp mkdir cd touch abc ectool --server localhost login admin changeme ectool publishArtifactVersion --version 1.0 --artifactName hello:world 4. Verify that the `/tmp/` is published to the Amazon S3 backing store. ![s3amazon verify backing store](_images/s3amazon-verify-backing-store.88dd6e0.png) Figure 2. Amazon S3 backing store example #### [](#_configuring_the_artifact_repository_server_for_google_cloud_platform)Configuring the artifact repository server for Google Cloud Platform You can configure the repository server to use a GCP storage bucket as the backing store. Before starting this procedure, make sure the following have been created and are available on the Google Cloud Platform ([https://console.cloud.google.com/](https://console.cloud.google.com/)): * A GCP storage bucket for the backing store. * A service account with the appropriate IAM permissions or IAM roles that have access to the `gcs://*` storage bucket you want to use with the repository server. For example, the service account could have the **Storage Admin** role. Refer to the Google Cloud Storage documentation for more information on IAM permissions and roles that can be used to access the storage bucket. The instance running the repository server must be started with a service account that can access the storage bucket used for the backing store. Set up the artifact repository for GCP, by doing the following: 1. In the `/opt/cloudbees/sda/conf/repository/server.properties` file set `REPOSITORY_BACKING_STORE` to use the GCP storage bucket by changing REPOSITORY_BACKING_STORE=repository-data to REPOSITORY_BACKING_STORE=gcs:// 2. Restart the artifact repository server, by running: * As a `root` user: /etc/init.d/commanderRepository restart * If using systemd: sudo systemctl restart commanderRepository * You can also use the `ecconfigure` command to change the backing store. This automatically restarts the repository server. ecconfigure --repositoryStorageDirectory gcs://test-repository-backingstore 3. Test the `REPOSITORY_BACKING_STORE` using the steps in Step 3 of [Configuring the artifact repository server for Amazon S3](#Configure-artifact-repository-server-Amazon-S3). 4. Verify that `/tmp/` is published to the GCP storage bucket. #### [](#_repositorydisabled_intrinsic_property)`repositoryDisabled` Intrinsic Property The repository object has a `repositoryDisabled` property that dictates whether artifacts can be published or retrieved from a particular repository. If you take a repository offline for maintenance, you can disable the repository in the CloudBees CD/RO server rather than shutting down the repository machine or the service. Steps that attempt to publish to this repository fail, and steps that attempt to retrieve artifact versions skip this repository during the search for available repositories. [](#Access_Control)Configuring access control --------------------------------------------- You can apply access control permissions to artifacts and artifact versions to control who can create an artifact, publish an artifact, read an artifact version, and retrieve an artifact version. Action Required permission Object Create an artifact Modify Artifacts system object Read artifact metadata Read The relevant artifact Modify an artifact Modify The relevant artifact Delete an artifact, which deletes all artifact versions within the artifact Modify Artifacts system object Publish an artifact version Modify The artifact into which a new version is being published Read artifact version metadata (required for retrieval) Read The relevant artifact version Modify an artifact version Modify The relevant artifact version Delete an artifact version Modify The artifact that owns the artifact version The **Execute** privilege does not take part in artifact or artifact version access control. By default, an **** access control entry is available that allows anyone to publish an artifact version to any artifact and allows anyone to create an artifact. You might consider removing this access control entry and setting up tighter control for individual artifacts. Refer to the EC-Security plugin for help with this effort. For more information about access control for artifact management, refer to [Access control](/docs/cloudbees-cd/latest/access-control/). The following scenarios provide ideas for how you might set up access control for artifact management. ### [](#_control_who_can_create_new_artifacts_and_publish_artifact_versions)Control who can create new artifacts and publish artifact versions CloudBees CD/RO administrator, Adam, received communication from development manager, Max, requesting that only he should be able to create artifacts in CloudBees CD/RO — developers of CloudBees CD/RO procedures under him should only be able to create steps to publish artifact versions under pre-defined artifacts. His reasoning is that it is too easy for a developer to introduce a typo during publish (which does auto-create the artifact if the user has the appropriate privilege and the artifact doesn’t exist). Adam accommodates the request by giving Max the modify privilege on the artifacts system object and removing the `` access control entry. Max then creates artifacts and assigns modify privileges to those artifacts for his development group, effectively allowing them to publish artifact versions. At run-time, a step belonging to some project actually performs the publishing, so Max adds project principal access control entries for the relevant projects as well, providing modify privileges. ### [](#_control_which_projects_can_retrieve_artifact_versions_at_run_time)Control which projects can retrieve artifact versions at run time Acme Corporation’s router product has a software stack consisting of OS, PlatformLibraries, and Apps. Each of these has a CloudBees CD/RO project with several procedures that work together to build the relevant component. PlatformLibraries builds against some of the C header files and libraries produced by the OS build. Similarly, Apps builds against libraries produced by PlatformLibraries. The lead developer for the OS component, creates a `Router:OSLib` artifact and gives the PlatformLibraries project principal read privileges so that project can retrieve any OSLib artifact version it needs to build PlatformLibraries. Similarly, Paul from the PlatformLibraries team creates a `Router:PlatformLib` artifact and gives the Apps project principal read privileges. ### [](#_group_1_publishes_an_artifact_version_and_wants_only_group_2_to_be_able_to_retrieve_it)Group 1 publishes an artifact version and wants only Group 2 to be able to retrieve it An organization wants to create artifacts for each of the third-party packages used by developers. Among the tools shared is the GWT package. The build manager chooses to create an artifact called `3rd-party-PKGS:GWT`. Because of licensing concerns, the build team must control or regulate who has access to the GWT and who can publish artifact versions used within the company’s product. In this case, the build team publishes a GWT artifact version and tests it. During the test and approval cycles, only the build team has access to read and modify the artifact or publish new artifact versions. After a version has passed internal testing and approvals, the build team then grants read access to each of the developer groups. [](#Publishing_Artifact_Versions)Publishing artifact versions ------------------------------------------------------------- An artifact version is published by using a procedure step or the _ectool /_ _cb-perl_ command-line interface. In most cases, using the Publish Artifact Version step is sufficient. For more information on publishing artifact versions, refer to [Artifact versions](/docs/cloudbees-cd/latest/artifacts/artifact-versions). [](#Enable_Compression)Enabling compression ------------------------------------------- When publishing an artifact, consider whether you should enable compression. Using compression reduces transfer time during publishing. However, compression also adds overhead when computing the compressed data. If files included in the artifact version are primarily text files or are another highly compressible file format, the benefit of reduced transfer time outweighs the cost of computing compressed data. Artifact versions that contain installers, jars, audio, and video are almost certainly **not** compressible because these file types are already compressed, so the cost of compressing outweighs the (near zero) benefits of reducing transfer time. Another consideration: Artifact versions are stored in the same format as they are transferred, so highly compressible artifact versions use less space in the repository backing store. [](#Include___Exclude_patterns)Using include and exclude patterns ----------------------------------------------------------------- When publishing an artifact, you can choose which files to include by using include and exclude patterns. File patterns are expressed as relative paths under the From Directory. Pattern syntax and behavior is the same as Ant and uses the following wildcard characters: `?` — matches a single character `*` — matches any number of characters, but only at a single directory level `**` — matches any number of directory levels Examples: Use `*.jar` to match any `.jar` file in the top-level directory. Use `*/*.jar` to match any `.jar` file in any child directory. Use `**/*.jar` to match any `.jar` file at any level. [](#retrieving_artifact_versions)Retrieving Artifact Versions ------------------------------------------------------------- An artifact version is retrieved using a procedure step, or the `retrieveArtifactVersion` API call, used by the ectool/cb-perl command-line tools. This API call returns the most current artifact version that meets your criteria, per the algorithm described below. Typically, you specify a version number range. Artifact version ranges are specified using interval notation. Brackets `[]` indicate versions to include and parentheses `()` indicate versions to exclude. For example, the following artifact versions are published: foobar:test:3.0.1-3645 foobar:test:1.2.3-2139 foobar:test:2.0.0-4000 foobar:test:2.0.0-DE-3395 foobar:test:2.0.0-DE-2445 foobar:test:2.0.0-DE foobar:test:3.0.0-3539 foobar:test:4.0.0-5584 You want to retrieve the most current artifact version for the `foobar:test` artifact. To do this, you do not need to provide a version or version range, but instead enter retrieve parameters in the Retrieve Artifact Version step page as follows: ![retrieve current version](_images/retrieve-current-version.a825d17.png) Figure 3. Retrieve artifact version parameters The equivalent command using `ectool` is: ectool retrieveArtifactVersions --artifactName foobar:test and with cb-perl, the syntax is: my $cmdr = new ElectricCommander(); $cmdr->retrieveArtifactVersions({artifactName => "foobar:test"}); Version numbers for two artifact versions (Artifact Version 1 and Artifact Version 2) are compared in the server as follows: 1. Compare `major.minor.patch`. 2. If they are equal, compare `qualifiers`. 3. If they are equal, compare `buildNumbers`. This comparison algorithm does not take the time of publish into account. "Current" means the artifact version with the greatest version per the above algorithm. The above list of artifact versions is sorted (most current to least) as follows: * `foobar:test:4.0.0-5584` * `foobar:test:3.0.1-3645` * `foobar:test:3.0.0-3539` * `foobar:test:2.0.0-DE-3395` * `foobar:test:2.0.0-DE-2445` * `foobar:test:2.0.0-DE` * `foobar:test:2.0.0-4000` * `foobar:test:1.2.3-2139` To retrieve the latest 3.0.0 artifact version, set the maximum version to 3.0.1 "exclusive". The parameters entered for the retrieve step look like this: ![retrieve current version exclusive](_images/retrieve-current-version-exclusive.113d257.png) Figure 4. Retrieve artifact version parameters Setting this constraint ensures the entire "version string" is properly evaluated. In other words, you want to ensure all artifact versions with 3.0.0, including qualifiers and build numbers, are included in the evaluation. The equivalent command using `ectool` is: ectool retrieveArtifactVersions --artifactName foobar:test --versionRange "(,3.0.1)" And with cb-perl, the syntax is: my $cmdr = new ElectricCommander(); $cmdr->retrieveArtifactVersions({artifactName => "foobar:test", versionRange => "(,3.0.1)"}); Note the use of parentheses to exclude version "3.0.1". Generally, while in development, you want to specify a minimum artifact version and take the latest version published, then when an artifact has reached a "released" state, indicate a specific version. ### [](#_parameters_and_environment_variables_for_artifact_retrieval_retry_attempts)Parameters and environment variables for artifact retrieval retry attempts The following parameters and their corresponding environment variables for artifact retrieval retry attempts are available in the Automation Platform server settings: * **Number of retries for Retrieve Artifact** and `COMMANDER_NUMBER_OF_RETRIES_FOR_RETRIEVE_ARTIFACT`: Number of retries when doing artifact retrieval. The default is three retries. * **Initial delay between retries for Retrieve Artifact** and `COMMANDER_INITIAL_DELAY_BETWEEN_RETRIES_FOR_RETRIEVE_ARTIFACT`: Initial delay (in milliseconds) between retries when doing artifact retrieval using backing off algorithm. Defaults to 100 ms. * **Maximum delay between retries for Retrieve Artifact** and `COMMANDER_MAXIMUM_DELAY_BETWEEN_RETRIES_FOR_RETRIEVE_ARTIFACT`: Maximum delay (in milliseconds) between retries when doing artifact retrieval using backing off algorithm. Defaults to 30000 ms. You set the variables on a client machine before doing a manual retrieval from the client via the `ectool retrieveArtifactVersions` command. [](#Supplying_filters_during_a_retrieve_operation)Entering filters during a retrieve operation ---------------------------------------------------------------------------------------------- At times, a company might want to distinguish between artifact versions not just by version range but also by some other property of the artifact version. Examples are: * Specifying a qualifier in the version string when publishing. For example, you might want to publish an artifact version for your product’s English or German variants. You could publish the English version as `2.0.4-EN-55497` and the German version as `2.0.4-DE-55497`. When the qualifier string is used this way to classify artifact versions, it should be used as a filter to retrieve the latest version. This is described in the following example. * Setting a custom property on the artifact version. For example, after your QA team has tested an artifact version, they could show approval by setting a `qaApproval` custom property, including the name of the QA engineer. For either case, filters can be applied when retrieving an artifact version. To retrieve the latest German version for the artifact, the parameters entered on the retrieve artifact custom step page look like this: ![retrieve current version exclusive](_images/retrieve-current-version-exclusive.113d257.png) Figure 5. Retrieve artifact version parameters Because `DE` was used as the qualifier, CloudBees CD/RO retrieves the latest artifact version for the `perl:pkg` artifact where the qualifier equals `DE`. Only the following filter operators can be applied to the intrinsic property version: * `equals` * `greater than` ( `greaterThan`,for use in the API) * `less than` ( `lessThan`, for use in the API) * `greater than or equals` ( `greaterOrEqual`, for use in the API) * `less than or equals` ( `lessOrEqual`, for use in the API) `equals` is special because it does a string comparison on the version string of the artifact version, while the other operators compare the interpretation of the version string as described earlier. All filter operators can be applied to version string components: `majorMinorPatch`, `qualifier`, or `buildNumber`. To find artifact versions with no qualifier, specify a filter on "qualifier" with operator "is not set" (or `isNull` in the API). For the QA approval case in a previous example, if a procedure is interested in an artifact version that was approved by QA, the retrieve step specifies a filter for `qaApproval is set`. `ectool` does not currently support the filters argument, but you can write a Perl script as follows: my $cmdr = new ElectricCommander(); $cmdr->retrieveArtifactVersions({artifactName => "perl:pkg", filters => {propertyName => "qaApprover", operator => "isNotNull"}}); [](#Dependent_artifact_version_s)Entering dependent artifact versions --------------------------------------------------------------------- A published artifact version can be dependent on a list of artifact versions. These dependent artifact versions are retrieved when the primary artifact version is retrieved, and they are specified with a query syntax when publishing the primary artifact version. For example, an artifact version for the CloudBees CD/RO product could include dependent artifact versions for the core product, the SDK, and online Help: * The latest `commander:Core` artifact version greater than or equal to version 3.5 (including 3.5) * The most recent version of the `commander:SDK` artifacts * The `commander:onlineHelp` artifact with a version greater than version 3.10 (excluding 3.10) Because dependent artifact versions are evaluated during the retrieval process, specify them using the same syntax as for retrieving any artifact version. This is done by including the group ID, the artifact key, and a specific artifact version or a version range. Parameters for the "publish" custom step page are similar to the following: ![retrieve artifact version](_images/retrieve-artifact-version.94cde27.png) Figure 6. Retrieve artifact version parameters A published artifact version could have zero files, that is, no files are published as part of the artifact version, but it still could have a list of dependent artifact versions. An empty artifact version might be useful as a modular convenience container for retrieving a group of artifact versions in downstream processes. [](#Understanding_Artifact_Usage)Understanding artifact usage ------------------------------------------------------------- CloudBees CD/RO has an artifact repository that can store build outputs as artifacts and their versions. These artifacts are then used in component definitions for application models; they can be used in Master component definitions as shared components. And when applications are deployed, these artifacts are deployed to various environments and their specific resources. You should understand various use cases in which a specific artifact version is used. The following are a few examples. ### [](#_example_performing_vulnerability_analysis_of_an_artifact)Example: Performing vulnerability analysis of an artifact Suppose that an artifact version is used in various application models and master components and that the artifact is deployed to various environments. Also suppose that it is discovered that the particular artifact version has a security vulnerability. In such cases, it might be important to quickly know the usage of that artifact version so that corrective action can be taken. ### [](#_example_understanding_dependencies)Example: Understanding dependencies Suppose users have created application components using a copy of a Master component. If the definition of that Master component changes, you should understand the dependency and impact so that application owners who leverage the Master component can take corrective action. You can use the Get Artifact Usage procedure in the CloudBees CD/RO to understand how and where an artifact and component is used to perform vulnerability analysis or to understand dependencies or any other artifact usage use case. When this procedure is run, it asks for inputs such as a component artifact name and version as well as an email address for sending out a usage report. ![get artifact usage](_images/get-artifact-usage.65e8b25.png) Figure 7. Get artifact usage For example, suppose that you want to generate a report for the usage of an artifact named Shared:Login with version 4.2 and then email the report to a user. You can enter that information and run the procedure. This generates a report that shows clearly which master components, applications, and environments have links to the 4.2 version of the Shared:Login artifact. ![links shared artifact](_images/links-shared-artifact.5e31059.png) Figure 8. Shared artifact Users can easily understand the current usage of an artifact or a master component, and the system emails this report to the respective users. This procedure can be extended easily to take corrective actions if needed, such as redeploying a new version of the artifact to all the current environments to remove the security vulnerability. [](#Artifact_cache)Understanding the artifact cache --------------------------------------------------- Artifact versions are retrieved to an artifact cache directory before being consumed by a build process. Properly configured, the artifact cache directory can significantly improve efficiency as you work with artifacts. The artifact cache directory is set in one of these ways: * Enter information in the Artifact Cache Directory field on the Edit Resource page. * Use the `modifyResource` API to set the `artifactCacheDirectory` intrinsic property. * Set the `artifactCache` field in the `agent.conf` file. Thus, each resource can have its own artifact cache location (directory). Alternatively, multiple resources and an entire site can share an artifact cache if the cache directory resides on a shared file server. When a CloudBees CD/RO step retrieves an artifact version, remember that often a version range or no version at all is specified. For example, to retrieve the latest CloudBees CD/RO SDK artifact version, you could issue the following command: ectool retrieveArtifactVersions --artifactName commander:SDK No version is specified because we want the "latest" artifact version published. In the following example, the step sends a request (1) to the CloudBees CD/RO server: 1. The CloudBees CD/RO server replies (2) with a fully qualified artifact version name. In this case, it returns `commander:SDK:1.2.0.43552`. 2. The step then looks in its artifact cache for the artifact version (3). 3. If the artifact is not found in the artifact cache, a request is issued (4) to get a list of repositories to search to find the artifact version. 4. The CloudBees CD/RO server replies (5) with a list of repositories and the order in which to search them. 5. The step requests the artifact version from the first (6) repository in the list. 6. If the artifact version is found in the repository, the artifact version is retrieved to the cache (7). 7. If the artifact version is not found, the step requests the artifact version from the next repository in the search order (8). 8. If the artifact version is found in that repository, the artifact version is retrieved to the cache (9). 9. Otherwise, if the artifact version is not found in any repository, an error is displayed. ![artifact version not found](_images/artifact-version-not-found.068a69c.jpg) Figure 9. Artifact version not found In this example, if the requested artifact version was found in the artifact cache (step 3), the resource would not have communicated with either of the repository servers. Also, any potential network latencies would have been avoided from retrieving the artifact version from the repository server. If you have a cache shared by multiple resources and steps that run on those resources need to use the same artifact version, the first step requesting an artifact version retrieves it from the repository to the cache. Later job steps using the artifact version have immediate access to the artifact version in the cache, eliminating the need for each step to retrieve its own copy of an artifact version. [](#Cleaning_up_repositories_and_caches)Cleaning up repositories and caches --------------------------------------------------------------------------- This method does not work for artifacts in an Amazon Simple Storage Service (Amazon S3) repository. Deleting an artifact version in the CloudBees CD/RO server does not delete it in the repository backing store, and it does not remove the artifact version from an artifact cache. _ectool_ and the CloudBees CD/RO Perl module include a function for each of these tasks: `cleanupRepository` and `cleanupArtifactCache`. Both of these API calls are intended to be run on the machine containing the `cache/backingstore` directory that needs to be cleaned up. The cleanup algorithm works as follows: 1. Review the directory tree looking for directories three-levels deep. 2. For each such directory, interprets the three path components as < `groupId`, `artifactKey`, `version` >. 3. Issues a `findObjects` request to the CloudBees CD/RO server to check whether an artifact version with that specification exists. 4. Deletes all directories identified by < `groupId`, `artifactKey`, `version` > and not recognized by the CloudBees CD/RO server. To operate correctly, using the `cleanupRepository` API _requires_ read access to all artifact version objects stored in CloudBees CD/RO and therefore must be run in an "admin" session. CloudBees also recommends running `cleanupArtifactCache` in an "admin" session—if not, artifact versions that are simply not readable by the user invoking the cleanup are deleted from the cache. In addition, both functions fail if the server does not recognize any of the chosen artifact versions. Thus, if the cleanup function is called on a directory that is not a `backingstore` directory or and artifact cache, it does not delete anything. The recommended approach for removing stale artifact versions from a backing store is to use a CloudBees CD/RO maintenance procedure that periodically runs on each repository server (that presumably has an agent installed). Similarly, use a procedure with a broadcast step to run on each agent to clean up its cache. The job step session may not have read privileges on all artifact versions. CloudBees recommends that you sign in as a privileged user, and then perform cleanup with those credentials.