Artifact management overview

During the development process, project components or "outputs" are produced for consumption by other projects or the main project to create a whole entity. Applications are assembled by combining these various outputs (dependencies) and packaging them together into a deployable software application.

Outputs can be libraries, scripts, graphics, and so on. This output is an artifact that can be consumed by other projects.

Every stage in the application lifecycle produces and consumes artifacts. Artifacts are critical to the build-test-deploy process.

Using artifacts can:

Improve performance across builds.
Provide better reusability of components.
Improve cross-team collaboration with greater traceability.

For example, instead of each developer repeatedly downloading third-party packages from external sources, these components can be published and versioned as an artifact. A developer then simply retrieves a specific artifact version from a local repository, which guarantees a consistent package from build to build.

About artifact objects

CloudBees CD/RO has three types of objects to support Artifact Management functionality: artifact, artifact version, and repository. Similar to other CloudBees CD/RO objects, each of these objects support custom properties and access control.

Artifact

An artifact is a top-level object containing artifact versions, a name template for published artifact versions, artifact specific properties, and access control entries to specify privileges. If an artifact is deleted, all published artifact versions within that artifact are also deleted.

You can create artifacts in several ways:

Use the New Artifact web page
Use the createArtifact API command with the command-line tool (ectool) or an cb-perl script
An artifact is created when you publish an artifact version if the artifact did not already exist, and the user issuing the publish command has permission to create artifacts. Refer to Configuring access control and Publishing artifact versions for more information.

To create an artifact, you enter a Group ID and Artifact Key. Together, these create an artifact name in the form groupId:artifactKey. You cannot modify the name after CloudBees CD/RO creates the artifact name from your supplied Group Id and Artifact Key specifications.

Plan your artifact Group IDs and Artifact Keys carefully to prevent the extra work of deleting a name and beginning your specifications again. The use of a groupId and artifactKey allow two namespaces for added flexibility in organizing artifacts within your company or organization.

Examples

If your development team uses GWT for web development and JUnit for unit testing, you may want to create ThirdParty:GWT and ThirdParty:JUnit artifacts.
Your organization may have two different development teams (for example, OS and Apps) who want to use artifacts. In building each product, both development groups want to publish and use an SDK with their product, but each SDK is different and unique. One solution is to create one artifact named os:SDK and another artifact named apps:SDK. By using different Group IDs, each development team is free to name their artifacts however they choose, without worrying about name collisions with artifacts produced by the other team.

To reference an artifact, you can use the groupId and artifactKey combination or simply use the artifactName. The /artifacts/<artifactName> property path allows you to access any properties of a given artifact.

Artifact version

An artifact version is a collection of 0 to N files that were published to an artifact repository. Artifact version metadata is stored in an artifactVersion object in the CloudBees CD/RO server.

Tips for working with artifact versions:

To group a collection of files (typically from build output in a CloudBees CD/RO workspace) into an artifact version, specify the "from" directory containing those files.
To include files from multiple directories in your artifact, you need to specify a common root directory to include these directories when publishing your artifact.
To restrict which files and subdirectories in the "from" directory to include in the artifact version, you can specify "include" and "exclude" patterns.
Artifact versions are stored on an artifact repository server in either uncompressed tar archives or compressed tar-gzip archives.
Artifact versions can be created using the publishArtifactVersion API, which includes creating the object in the CloudBees CD/RO server and publishing the artifact version to the artifact repository.

Interactively creating an artifact version using the automation platform web UI is not supported.
When you retrieve an artifact version, the original directory structure of the directory where files were published "from" is preserved.
When you retrieve/publish artifacts, directories containing symlinks are always resolved; they are not preserved.

Artifact versions are referenced by Group ID, Artifact Key (or artifact name) and a version string.

The artifact version’s name is set based on the name template on the "owning" artifact. By default, the name template is in the form groupId:artifactKey:version.
Each artifact version name must be unique within a CloudBees CD/RO server installation.

The property path /artifactVersions/<artifactVersionName> allows access to any properties for a particular artifact version. Because you may not know the artifact version name (because of the name template), this path supports the form <groupId>:<artifactKey>:<version> as a substitute for the name of the artifact version.

All CloudBees CD/RO API’s that take an artifactVersionName argument also accept this alternate form as a substitute, similar to how other APIs that accept a jobId argument accept the job name as a substitute.

Version strings

A version string must be provided when publishing an artifact version.

For example:

5.8.8-EN-55842 or 5.8.8.55842-EN: In either form, the version string is interpreted as:

Major version number: 5

Minor version number: 8

Patch level number: 8

Qualifier: EN

Build number: 55842

Leading zeros are not valid for the major, minor, and patch versions.

You must provide separator punctuation. When interpreting (parsing) the first version string form, CloudBees CD/RO interprets the text after the first hyphen as the build number if this string contains numeric characters only. Otherwise, the string is interpreted as the qualifier.

Version string examples and how CloudBees CD/RO interprets them:

Input (version string) Major Minor Patch Qualifier Build Number

	Input (version string)	Major	Minor	Patch	Qualifier	Build Number
1	`2.0.3-44834`	`2`	`0`	`3`	`<empty/null>`	`44834`
2	`3.8.4-RC1`	`3`	`8`	`4`	`RC1`	`0`
3	`3.8.4-RC1-36`	`3`	`8`	`4`	`RC1`	`36`
4	`3.8.4.36`	`3`	`8`	`4`	`<empty/null>`	`36`
5	`1-36`	`1`	`0`	`0`	`<empty/null>`	`36`
6	`3.8.4.36-RC1`	`3`	`8`	`4`	`RC1`	`36`

2.0.3-44834

2

0

3

<empty/null>

44834

3.8.4-RC1

3

8

4

RC1

0

3.8.4-RC1-36

3

8

4

RC1

36

3.8.4.36

3

8

4

<empty/null>

36

1-36

1

0

<empty/null>

36

3.8.4.36-RC1

3

8

4

RC1

36

In this table:

Row 5 shows that you are not required to specify all three of <major, minor, patch> when publishing an artifact version if your organization’s versioning conventions do not use all three fields. However, there is a caveat with this level of flexibility. You can create two artifact versions in the system with equivalent versions. Version string "1-36" is equivalent to "1.0-36", but they are not the same, so you can publish two artifact versions with these version strings successfully. When retrieving an artifact version, in some cases it could be unclear as to exactly which one you get. For this reason, CloudBees recommends that you adopt a convention and abide by it for a particular artifact.

The artifact version object exposes components of the version string by using these intrinsic properties:

version: The version component is the combination of major.minor.patch-qualifier-buildNumber. A version component does not need to include all the following intrinsic properties. Refer to the table above.
majorMinorPatch: This is the major.minor.patch version component as specified in the version string. For example, if you specified a version string of "1-36", the majorMinorPatch component is "1", not "1.0.0", which makes it possible for you to meet your naming conventions by reconstituting the version string a different way in the artifact version name template.
qualifier: The qualifier component of the version.
buildNumber: The build number component of the version.

The `artifactVersionState` property

Artifact versions have an artifactVersionState property whose value can be one of the following:

publishing: The artifact version is currently being published and is not available for retrieval.

available: The artifact version is available for retrieval if needed.

unavailable: The artifact version is not available for retrieval while in this state.

You can manipulate the state between available and unavailable on the Edit Artifact Version web page. This action can be useful if an artifact version seems to be corrupt in some way, but you want to investigate before potentially deleting it from the system.

By making the artifact version unavailable, retrievers can acquire an older (potentially more stable) artifact version while you look into the problem. If the artifact version is good, you can simply make it available again.

Artifact repository

The artifact repository is a machine where artifact versions are stored. The repository server is configured to store artifact versions in a directory referred to as the repository backing store.

By default, the backing store is the <data_dir>/repository-data directory in the repository installation. You can change this default setting by running ecconfigure --repositoryStorageDirectory on the repository server. The repository server listens on port 8200 for HTTPS requests to publish and retrieve artifact versions.

You cannot provide a mapped drive path when defining a backing store. For example, the path in the following command is not allowed and will cause artifact publishing to fail:

ecconfigure --repositoryStorageDirectory c:/repository-data▼

You must enter a UNC path instead. For example, the path in the following command is allowed:

ecconfigure --repositoryStorageDirectory //10.0.109.72/repo_share/repository-data▼

Connection information is stored in the repository object on the CloudBees CD/RO server.

You can create a repository objects in several ways:

Use the New Repository web page in the CloudBees CD/RO UI
Use createRepository API
Repository objects can be created automatically during the CloudBees CD/RO installation.

When you are installing a repository server , the installer creates a corresponding repository object on your CloudBees CD/RO server. If you are installing a repository server on the same machine as your CloudBees CD/RO server, a repository object named "default" is automatically created. If you are installing a repository server on a different machine than your CloudBees CD/RO server, the installer prompts you for information it uses to create the corresponding repository object .

When installing a repository server, you should install an agent also to ease maintenance tasks such as clearing out stale artifact versions from the repository backing store.

Select the Artifacts tab and then the Repositories subtab to view your list of created repositories.

List of created repositoriesrole="screenshot"

Figure 1. List of created repositories

You can re-order the repository search order (for retrieving artifact versions). To do so, use the mouse to grab and drag the icon in the left column to the row position you prefer, effectively re-ordering the repository list.

For distributed environments, where the preferred repository search order varies depending on which resource is performing the retrieval, you can specify a preference order on the Edit Resource panel.

Figure 2. Preference order

The Gateway(s): field that appears in the Resource Details panel is not editable (not even for a gateway resource) and therefore does not appear in the Edit Resource panel. This field is automatically filled and is only for gateway agents. For example, in the Resource Details panel for agentp1-gw-01 and agentp2-gw-01 resources as they form a gateway Default_to_zone1_gateway using the steps in the KBEC-00393 - Setting up an environment with two zones and a gateway KB article.

To go to the Edit Resource panel: from the Repositories page, select the Cloud tab to open the Resources page, then select a resource name. Enter one repository name per line in the Repository Names field.

In the following example, a resource in California (A1) uses the "default" repository search order (that is, the order on the Repositories page). A1 searches for artifact versions by first requesting an artifact version from Repository 1. If the artifact version requested is not found, it then requests the artifact version from Repository 2.

The resource in New York (A2) has a specific search order specified where it searches for artifacts from Repository 2, first, then searches Repository 1.

Figure 3. Repository search order

Using multiple repositories

CloudBees CD/RO supports multiple repositories, however you cannot share a repository between multiple CloudBees CD/RO servers. Multiple repository servers may be required in your organization because of business or organizational reasons, government and compliance requirements, or for performance and data protection.

For example:

Business or organizational structure: Within a company, two different development teams want to have artifact repositories containing artifacts and artifact versions specific to their product. A repository sever is created for each team. The two teams publish artifact version to their respective repositories.
Performance: A company has multiple development sites. Instead of having a single repository server, which creates significant network performance issues from retrieving artifact versions during the build process, they create a repository at each development location and have artifact versions synchronized between the repository servers. In this way, when artifact versions are retrieved for use by a build process, they are retrieved from a local repository.
Data protection and disaster recovery: A company has a single CloudBees CD/RO server with two artifact repository server machines. The contents of the repository servers are synchronized and replicated between both machines. In the event repository 1 is unavailable during a retrieve, repository 2 is searched for the requested artifact retrieval.

Copying artifacts to multiple repositories

You can synchronize artifacts from a source artifact repository to one or more target repositories. This helps to support distributed architectures with multiple CloudBees CD/RO artifact repository servers. You must specify the exact artifact and its version as input. This feature uses the EC-Artifact plugin, which is bundled with CloudBees CD/RO.

To synchronize artifacts to other repositories, select Administration Plugins EC-Artifact and select Run to run the SyncArtifactVersion procedure. The following dialog appears:

Figure 4. All repositories

Entering parameters

Enter the following parameters as needed.

Parameter Description

Parameter	Description
Artifact	Artifact name. The format is `<groupId>:<artifactKey>`.
Version	Full artifact version string. The format is `<major>.<minor>.<patch>-<qualifier>-<buildNumber>`. For example, `5.8.8-EN-55842`. The version string must be unique across all of this artifact’s versions. For more information, refer to Version_Strings.
Source Repository	Name of the repository from which this artifact version is retrieved.
Target Repository	Name of the repository where the artifact is copied. You can add more repositories by selecting Add target repository .
Overwrite?	(Optional) Specifies whether to overwrite any existing artifact with the same name and version on the target repositories. This option is disabled by default.
Upload in parallel?	(Optional) Specifies whether to copy artifacts to multiple target repositories in parallel (for faster performance) instead of serially. You should choose whether to use this option based on how the number of target repositories and the artifact size could impact network traffic. This option is disabled by default.
Priority	(Optional) Priority of the job. The choices are low, normal, high, or highest.
Impersonation	(Optional) Credential impersonation. The choices are: Use pre-defined credential: Uses the predefined credential. This is the default. Use specific credential: You must specify a project name and a credential name. Use a specific user: You must specify a user name under which you want to log in and the password for the user name to be used for LDAP testing.

Artifact

Artifact name. The format is <groupId>:<artifactKey>.

Version

Full artifact version string. The format is <major>.<minor>.<patch>-<qualifier>-<buildNumber>. For example, 5.8.8-EN-55842.

The version string must be unique across all of this artifact’s versions. For more information, refer to Version_Strings.

Source Repository

Name of the repository from which this artifact version is retrieved.

Target Repository

Name of the repository where the artifact is copied. You can add more repositories by selecting Add target repository .

Overwrite?

(Optional) Specifies whether to overwrite any existing artifact with the same name and version on the target repositories. This option is disabled by default.

Upload in parallel?

(Optional) Specifies whether to copy artifacts to multiple target repositories in parallel (for faster performance) instead of serially. You should choose whether to use this option based on how the number of target repositories and the artifact size could impact network traffic. This option is disabled by default.

Priority

(Optional) Priority of the job. The choices are low, normal, high, or highest.

Impersonation

(Optional) Credential impersonation. The choices are:

Use pre-defined credential: Uses the predefined credential. This is the default.
Use specific credential: You must specify a project name and a credential name.
Use a specific user: You must specify a user name under which you want to log in and the password for the user name to be used for LDAP testing.

Then click Run.

Understanding overwrite behavior

The following table provides an example that illustrates the overwrite behavior.

Repository	Result with Overwrite disabled	Result with Overwrite enabled
Repository 1 (source): Repository containing artifact1 to be copied	–	–
Repository 2 (target): Repository without artifact1	artifact1 is copied to Repo2	artifact1 is copied to Repository 2
Repository 3 (target): Repository containing artifact1	artifact1 on Repository 3 is not overwritten	artifact1 on Repository 3 is overwritten
Repository 4 (target): Repository that does not exist	Error	Error

Repository

Result with Overwrite disabled

Result with Overwrite enabled

Repository 1 (source): Repository containing artifact1 to be copied

–

Repository 2 (target): Repository without artifact1

artifact1 is copied to Repo2

artifact1 is copied to Repository 2

Repository 3 (target): Repository containing artifact1

artifact1 on Repository 3 is not overwritten

artifact1 on Repository 3 is overwritten

Repository 4 (target): Repository that does not exist

Error

Troubleshooting

The SyncArtifactVersion procedure finishes with an error in the following situations:

Repository is unreachable: The target repository might be down.
Insufficient access rights: Access rights for writing to the target repository might be missing.

Configuring the artifact repository server for Amazon S3

You can configure the repository server to use Amazon S3 as the backing store and connect to Amazon S3 using IAM roles. This section describes the changes to make in the /opt/cloudbees/sda/conf/repository/server.properties file.

Before starting this procedure, make sure the following have been created and are available on the AWS Management Console ( https://aws.amazon.com/console/ ):

An Amazon S3 bucket for the backing store.
The IAM role with a policy that grants all the s3:\* access to the Amazon S3 bucket you want to use with the repository server.

For example, the IAM role called s3repositoryFullAccess has a policy called s3fullaccessAndAccessToInstallers that grants all the s3:\* access to the backing store bucket called ec-test-repository-backingstore.

Follow these steps to set up the artifact repository for Amazon S3:

In the /opt/cloudbees/sda/conf/repository/server.properties file, set REPOSITORY_BACKING_STORE to use Amazon S3 by changing
```
REPOSITORY_BACKING_STORE=repository-data▼
```
to
```
REPOSITORY_BACKING_STORE=s3://<name of the backing store bucket>▼
```
You can get the <name of the backing store bucket> by navigating to IAM > Roles > <name of the IAM role> page on the AWS Management Console. Then select Show Policy to open the Show Policy window to view the name of the backing store bucket.

In the following example, <name of the backing store bucket> is ec-test-repository-backingstore:

Figure 5. Name of backing store bucket
Restart the artifact repository server by running:
```
sudo systemctl restart commanderRepository▼
```

To test the REPOSITORY_BACKING_STORE, create two files in a /tmp/<your username> directory, publish the directory, and verify that it is published using commands like this:

cd /tmp▼

mkdir <your username>▼

cd <your username>▼

touch abc▼

ectool --server localhost login admin changeme▼

ectool publishArtifactVersion --version 1.0 --artifactName hello:world▼

Verify that the /tmp/<your username> is published to the Amazon S3 backing store.

Figure 6. Verify publish

Configuring the artifact repository server for Google Cloud Platform

You can configure the repository server to use a GCP storage bucket as the backing store.

Before starting this procedure, make sure the following have been created and are available on the Google Cloud Platform ( https://console.cloud.google.com/ ):

A GCP storage bucket for the backing store.
A service account with the appropriate IAM permissions or IAM roles that have access to the gcs://* storage bucket you want to use with the repository server.

For example, the service account could have the Storage Admin role. Refer to the Google Cloud Storage documentation for more information on IAM permissions and roles that can be used to access the storage bucket.

The instance running the repository server must be started with a service account that can access the storage bucket used for the backing store.

Follow these steps to set up the artifact repository for GCP:

In the /opt/cloudbees/sda/conf/repository/server.properties file set REPOSITORY_BACKING_STORE to use the GCP storage bucket by changing
```
REPOSITORY_BACKING_STORE=repository-data▼
```
to
```
REPOSITORY_BACKING_STORE=gcs://<name of the backing store bucket>▼
```
Restart the artifact repository server, by running:
- As a root user:
  /etc/init.d/commanderRepository restart
  ▼
- If using systemd:
  sudo systemctl restart commanderRepository
  ▼
- You can also use the ecconfigure command to change the backing store. This automatically restarts the repository server.
  ecconfigure --repositoryStorageDirectory gcs://test-repository-backingstore
  ▼
Test the REPOSITORY_BACKING_STORE: follow the steps in Step 3 of Configuring the artifact repository server for Amazon S3.
Verify that /tmp/<your username> is published to the GCP storage bucket.

The `respositoryDisabled` Intrinsic Property

The repository object has a repositoryDisabled property that dictates whether artifacts can be published or retrieved from a particular repository. If you take a repository offline for maintenance, you can disable the repository in the CloudBees CD/RO server rather than shutting down the repository machine or the service. Steps that attempt to publish to this repository fail, and steps that attempt to retrieve artifact versions skip this repository during the search for available repositories.

Configuring access control

You can apply access control permissions to artifacts and artifact versions to control who can create an artifact, publish an artifact, read an artifact version, and retrieve an artifact version.

Action	Required permission	Object
Create an artifact	Modify	Artifacts system object
Read artifact metadata	Read	The relevant artifact
Modify an artifact	Modify	The relevant artifact
Delete an artifact, which deletes all artifact versions within the artifact	Modify	Artifacts system object
Publish an artifact version	Modify	The artifact into which a new version is being published
Read artifact version metadata (required for retrieval)	Read	The relevant artifact version
Modify an artifact version	Modify	The relevant artifact version
Delete an artifact version	Modify	The artifact that owns the artifact version

Action

Required permission

Object

Create an artifact

Modify

Artifacts system object

Read artifact metadata

Read

The relevant artifact

Modify an artifact

Modify

The relevant artifact

Delete an artifact, which deletes all artifact versions within the artifact

Modify

Artifacts system object

Publish an artifact version

Modify

The artifact into which a new version is being published

Read artifact version metadata (required for retrieval)

Read

The relevant artifact version

Modify an artifact version

Modify

The relevant artifact version

Delete an artifact version

Modify

The artifact that owns the artifact version

The Execute privilege does not take part in artifact or artifact version access control.

By default, an <Everyone, modify> access control entry is available that allows anyone to publish an artifact version to any artifact and allows anyone to create an artifact. You might consider removing this access control entry and setting up tighter control for individual artifacts. Refer to the EC-Security plugin for help with this effort.

For more information about access control for artifact management, refer to Access control.

The following scenarios provide ideas for how you might set up access control for artifact management.

Control who can create new artifacts and publish artifact versions

CloudBees CD/RO administrator, Adam, received communication from development manager, Max, requesting that only he should be able to create artifacts in CloudBees CD/RO — developers of CloudBees CD/RO procedures under him should only be able to create steps to publish artifact versions under pre-defined artifacts.

His reasoning is that it is too easy for a developer to introduce a typo during publish (which does auto-create the artifact if the user has the appropriate privilege and the artifact doesn’t exist).

Adam accommodates the request by giving Max the modify privilege on the artifacts system object and removing the <Everyone, modify> access control entry. Max then creates artifacts and assigns modify privileges to those artifacts for his development group, effectively allowing them to publish artifact versions.

At run-time, a step belonging to some project actually performs the publish, so Max adds project principal access control entries for the relevant projects as well, providing modify privileges.

Control which projects can retrieve artifact versions at run time

Acme Corporation’s router product has a software stack consisting of OS, PlatformLibraries, and Apps. Each of these has a CloudBees CD/RO project with several procedures that work together to build the relevant component. PlatformLibraries builds against some of the C header files and libraries produced by the OS build. Similarly, Apps builds against libraries produced by PlatformLibraries.

The lead developer for the OS component, creates a Router:OSLib artifact and gives the PlatformLibraries project principal read privileges so that project can retrieve any OSLib artifact version it needs to build PlatformLibraries. Similarly, Paul from the PlatformLibraries team creates a Router:PlatformLib artifact and gives the Apps project principal read privileges.

Group 1 publishes an artifact version and wants only Group 2 to be able to retrieve it

An organization wants to create artifacts for each of the third-party packages used by developers. Among the tools shared is the GWT package. The build manager chooses to create an artifact called 3rd-party-PKGS:GWT. Because of licensing concerns, the build team must control or regulate who has access to the GWT and who can publish artifact versions used within the company’s product.

In this case, the build team publishes a GWT artifact version and tests it. During the test and approval cycles, only the build team has access to read and modify the artifact or publish new artifact versions. After a version has passed internal testing and approvals, the build team then grants read access to each of the developer groups.

Uploading and publishing artifacts from a local file system using the UI

You can use the Automation Platform UI to upload artifact files from a file system. This makes development and testing of new flows easier than other methods that use the EC-Artifact plugin and APIs. You can upload artifact files for publishing a new version of an existing artifact, or you create a new artifact and upload artifact files to publish the first version. You can upload a single file or a folder.

To enable this functionality, you must navigate to the Edit Server Settings page in the Automation Platform UI and select Allow Artifact Publish from UI. Note that this functionality is not supported in Internet Explorer or Safari.

Uploading files to publish a new version of an existing artifact

To upload artifact files for publishing a new version of an artifact that already exists:

In the Automation Platform, choose Artifacts > Artifacts.

The list of existing artifacts appears. For example:

Figure 7. List of existing artifacts
Select the artifact for which you want to upload a new version.

The Artifact Details page appears. For example:

Figure 8. Artifact details
Select Publish Artifact.

If this button is not visible, you might not have the proper permissions to create an artifact version.

The Publish Artifact Version page appears. For example:

Figure 9. Publish artifact version

Complete the page as follows:

Field name Description

Field name	Description
Version	Fully-qualified artifact version string. The version string must be of the form `<major>.<minor>.<patch>-<qualifier>-<buildNumber>`. The version string must be unique across all of this artifact’s versions.
Repository	Name of the repository where the new artifact version is published.
Enable Compression	Compress the artifact version before storing it in the repository. This option is selected by default.
Artifact Files	Browse files or folders for publishing to the repository: File Upload —Lets you choose a single artifact. Folder Upload —Lets you choose a folder that contains one or more artifacts. If you choose this option, you can add any combination of include or exclude patterns to filter files in or out: The browser does not allow the upload if a linked file or folder with a broken link is specified.
Add Dependent Artifact Version	(Optional) Dependent artifact version. You can enter one dependent artifact version per line, each in the form `<groupId>:<artifactKey>:<versionRange>`. All dependent artifact versions must exist for this artifact version to be retrieved. When this artifact version is successfully retrieved, its dependent artifact versions are retrieved also.

Version

Fully-qualified artifact version string. The version string must be of the form <major>.<minor>.<patch>-<qualifier>-<buildNumber>. The version string must be unique across all of this artifact’s versions.

Repository

Name of the repository where the new artifact version is published.

Enable Compression

Compress the artifact version before storing it in the repository. This option is selected by default.

Artifact Files

Browse files or folders for publishing to the repository:

File Upload —Lets you choose a single artifact.
Folder Upload —Lets you choose a folder that contains one or more artifacts. If you choose this option, you can add any combination of include or exclude patterns to filter files in or out:

The browser does not allow the upload if a linked file or folder with a broken link is specified.

Add Dependent Artifact Version

(Optional) Dependent artifact version. You can enter one dependent artifact version per line, each in the form <groupId>:<artifactKey>:<versionRange>.

All dependent artifact versions must exist for this artifact version to be retrieved. When this artifact version is successfully retrieved, its dependent artifact versions are retrieved also.

Select Publish.

The page that shows the list of artifacts with the updated artifact appears.

Creating a New Artifact and Uploading Files to Publish the First Version

To upload and publish the first version of an artifact that you create:

In the Automation Platform, choose Artifacts > Artifacts.

The list of existing artifacts appears. For example:

Figure 10. List of artifacts
Select Create Artifact.

The New Artifact page appears:

Figure 11. New artifacts
Select Publish Artifact Version.

The page expands to let you upload files from a file system to publish a version of the new artifact:

Figure 12. Upload files to publish new artifact version

Notice that the Name field is derived from the group ID and the key of the artifact version, which you enter in the next step.

Complete the page as follows:

Field name Description

Field name	Description
Group ID	Group ID of the artifact version.
Artifact Key	Artifact ID of the artifact version.
Description	Text that describes this object. CloudBees CD/RO does not interpret this text.
Artifact Version Name Template	Template used to determine the default names of artifact versions.
Publish Artifact Version
Version	Fully-qualified artifact version string. A full version string must be of the form `<major>.<minor>.<patch>-<qualifier>-<buildNumber>`. The version string must be unique across all of this artifact’s versions.
Repository	Name of the repository where the new artifact version is published.
Artifact Files	Browse files or folders for publishing to the repository: File Upload — Lets you choose a single artifact. Folder Upload — Lets you choose a folder that contains one or more artifacts. If you choose this option, you can add any combination of include or exclude patterns to filter files in or out: The browser does not allow the upload if a linked file or folder with a broken link is specified.
Enable Compression	Compress the artifact version before storing it in the repository. This option is enabled by default.
Add Dependent Artifact Version	(Optional) Dependent artifact version. You can enter one dependent artifact version per line, each in the form `<groupId>:<artifactKey>:<versionRange>`. All dependent artifact versions must exist for this artifact version to be retrieved. When this artifact version is successfully retrieved, its dependent artifact versions are retrieved also.

Group ID

Group ID of the artifact version.

Artifact Key

Artifact ID of the artifact version.

Description

Text that describes this object. CloudBees CD/RO does not interpret this text.

Artifact Version Name Template

Template used to determine the default names of artifact versions.

Publish Artifact Version

Version

Fully-qualified artifact version string. A full version string must be of the form <major>.<minor>.<patch>-<qualifier>-<buildNumber>. The version string must be unique across all of this artifact’s versions.

Repository

Name of the repository where the new artifact version is published.

Artifact Files

Browse files or folders for publishing to the repository:

File Upload — Lets you choose a single artifact.
Folder Upload — Lets you choose a folder that contains one or more artifacts. If you choose this option, you can add any combination of include or exclude patterns to filter files in or out:

The browser does not allow the upload if a linked file or folder with a broken link is specified.

Enable Compression

Compress the artifact version before storing it in the repository. This option is enabled by default.

Add Dependent Artifact Version

(Optional) Dependent artifact version. You can enter one dependent artifact version per line, each in the form <groupId>:<artifactKey>:<versionRange>.

All dependent artifact versions must exist for this artifact version to be retrieved. When this artifact version is successfully retrieved, its dependent artifact versions are retrieved also.

Select OK.

The page that shows the list of artifacts with the new artifact appears.

Publishing artifact versions

An artifact version is published by using a procedure step or the ectool / cb-perl command-line interface. In most cases, using the Publish Artifact Version step is sufficient.

The following screen illustrates the parameters section for creating the Publish Artifact Version step:

Figure 13. Publish artifact versions

Enabling compression

When publishing an artifact, consider whether you should enable compression. Using compression reduces transfer time during publish. However, compression also adds overhead when computing the compressed data. If files included in the artifact version are primarily text files or are another highly compressible file format, the benefit of reduced transfer time outweighs the cost of computing compressed data.

Artifact versions that contain installers, jars, audio, and video are almost certainly not compressible because these file types are already compressed, so the cost of compressing outweighs the (near zero) benefit of reducing transfer time. Another consideration: Artifact versions are stored in the same format as they are transferred, so highly compressible artifact versions use less space in the repository backing store.

Using include and exclude patterns

When publishing an artifact, you can choose which files to include by using include and exclude patterns. File patterns are expressed as relative paths under the From Directory. Pattern syntax and behavior is the same as Ant and uses the following wildcard characters:

? — matches a single character

* — matches any number of characters, but only at a single directory level

** — matches any number of directory levels

Examples:

Use *.jar to match any .jar file in the top-level directory.

Use */*.jar to match any .jar file in any child directory.

Use **/*.jar to match any .jar file at any level.

Retrieving Artifact Versions

An artifact version is retrieved using a procedure step, or the retrieveArtifactVersion API call, used by the ectool/cb-perl command-line tools. This API call returns the most current artifact version that meets your criteria, per the algorithm described below.

Typically, you specify a version number range. Artifact version ranges are specified using interval notation. Brackets [] indicate versions to include and parentheses () indicate versions to exclude.

For example, the following artifact versions are published:

foobar:test:3.0.1-3645

foobar:test:1.2.3-2139

foobar:test:2.0.0-4000

foobar:test:2.0.0-DE-3395

foobar:test:2.0.0-DE-2445

foobar:test:2.0.0-DE

foobar:test:3.0.0-3539

foobar:test:4.0.0-5584▼

You want to retrieve the most current artifact version for the foobar:test artifact. To do this, you do not need to provide a version or version range, but instead enter retrieve parameters in the Retrieve Artifact Version step page as follows:

Figure 14. Retrieve current artifact version

The equivalent command using ectool is:

ectool retrieveArtifactVersions --artifactName foobar:test▼

and with cb-perl, the syntax is:

my $cmdr = new ElectricCommander();
$cmdr->retrieveArtifactVersions({artifactName => "foobar:test"});▼

Version numbers for two artifact versions (Artifact Version 1 and Artifact Version 2) are compared in the server as follows:

Compare major.minor.patch.
If they are equal, compare qualifiers.
If they are equal, compare buildNumbers.

This comparison algorithm does not take the time of publish into account. "Current" means the artifact version with the greatest version per the above algorithm.

The above list of artifact versions is sorted (most current to least) as follows:

foobar:test:4.0.0-5584
foobar:test:3.0.1-3645
foobar:test:3.0.0-3539
foobar:test:2.0.0-DE-3395
foobar:test:2.0.0-DE-2445
foobar:test:2.0.0-DE
foobar:test:2.0.0-4000
foobar:test:1.2.3-2139

To retrieve the latest 3.0.0 artifact version, set the maximum version to 3.0.1 "exclusive". The parameters you enter in the retrieve step look like this:

Figure 15. Parameters for retrieve step

Setting this constraint ensures the entire "version string" is properly evaluated. In other words, you want to ensure all artifact versions with 3.0.0, including qualifiers and build numbers, are included in the evaluation.

The equivalent command using ectool is:

ectool retrieveArtifactVersions --artifactName foobar:test --versionRange "(,3.0.1)"▼

And with cb-perl, the syntax is:

my $cmdr = new ElectricCommander();
$cmdr->retrieveArtifactVersions({artifactName => "foobar:test", versionRange => "(,3.0.1)"});▼

Note the use of parentheses to exclude version "3.0.1".

Generally, while in development, you want to specify a minimum artifact version and take the latest version published, then when an artifact has reached a "released" state, indicate a specific version.

Parameters and environment variables for artifact retrieval retry attempts

The following parameters and their corresponding environment variables for artifact retrieval retry attempts are available in the Automation Platform server settings:

Number of retries for Retrieve Artifact and COMMANDER_NUMBER_OF_RETRIES_FOR_RETRIEVE_ARTIFACT: Number of retries when doing artifact retrieval. Defaults to 3 retries.
Initial delay between retries for Retrieve Artifact and COMMANDER_INITIAL_DELAY_BETWEEN_RETRIES_FOR_RETRIEVE_ARTIFACT: Initial delay (in milliseconds) between retries when doing artifact retrieval using backing off algorithm. Defaults to 100 ms.
Maximum delay between retries for Retrieve Artifact and COMMANDER_MAXIMUM_DELAY_BETWEEN_RETRIES_FOR_RETRIEVE_ARTIFACT: Maximum delay (in milliseconds) between retries when doing artifact retrieval using backing off algorithm. Defaults to 30000 ms.

You set the variables on a client machine before doing a manual retrieval from the client via the ectool retrieveArtifactVersions command.

Entering filters during a retrieve operation

At times, a company might want to distinguish between artifact versions not just by version range but also by some other property of the artifact version. Examples are:

Specifying a qualifier in the version string when publishing.

For example, you might want to publish an artifact version for your product’s English or German variants. You could publish the English version as 2.0.4-EN-55497 and the German version as 2.0.4-DE-55497.

When the qualifier string is used this way to classify artifact versions, it should be used as a filter to retrieve the latest version. This is described in the following example.
Setting a custom property on the artifact version.

For example, after your QA team has tested an artifact version, they could show approval by setting a qaApproval custom property, including the name of the QA engineer.

For either case, filters can be applied when retrieving an artifact version. To retrieve the latest German version for the artifact, the parameters entered on the retrieve artifact custom step page look like this:

Figure 16. Filters for artifact version

Because DE was used as the qualifier, CloudBees CD/RO retrieves the latest artifact version for the perl:pkg artifact where the qualifier equals DE.

Only the following filter operators can be applied to the intrinsic property version:

equals
greater than ( greaterThan,for use in the API)
less than ( lessThan, for use in the API)
greater than or equals ( greaterOrEqual, for use in the API)
less than or equals ( lessOrEqual, for use in the API)

equals is special because it does a string comparison on the version string of the artifact version, while the other operators compare the interpretation of the version string as described earlier.

All filter operators can be applied to version string components: majorMinorPatch, qualifier, or buildNumber. To find artifact versions with no qualifier, specify a filter on "qualifier" with operator "is not set" (or isNull in the API).

For the QA approval case in a previous example, if a procedure is interested in an artifact version that was approved by QA, the retrieve step specifies a filter for qaApproval is set.

ectool does not currently support the filters argument, but you can write a Perl script as follows:

my $cmdr = new ElectricCommander();
$cmdr->retrieveArtifactVersions({artifactName => "perl:pkg",
 filters => {propertyName => "qaApprover",
 operator => "isNotNull"}});▼

Entering dependent artifact versions

A published artifact version can be dependent on a list of artifact versions. These dependent artifact versions are retrieved when the primary artifact version is retrieved, and they are specified with a query syntax when publishing the primary artifact version.

For example, an artifact version for the CloudBees CD/RO product could include dependent artifact versions for the core product, the SDK, and online Help:

The latest commander:Core artifact version greater than or equal to version 3.5 (including 3.5)
The most recent version of the commander:SDK artifacts
The commander:onlineHelp artifact with a version greater than version 3.10 (excluding 3.10)

Because dependent artifact versions are evaluated during the retrieve process, specify them using the same syntax as for retrieving any artifact version, which is by including the group ID, the artifact key, and a specific artifact version or a version range. Parameters for the "publish" custom step page are similar to the following:

Figure 17. Parameters for publishing a custom step page

A published artifact version could have zero files, that is, no files are published as part of the artifact version, but it still could have a list of dependent artifact versions. An empty artifact version might be useful as modular convenience container for retrieving a group of artifact versions in downstream processes.

Understanding artifact usage

CloudBees CD/RO has an artifact repository that can store build outputs as artifacts and their versions. These artifacts are then used in component definitions for application models; they can be used in master component definitions as shared components. And when applications are deployed, these artifacts are deployed to various environments and their specific resources. You should understand various use cases in which a specific artifact version is used. Following are a few examples.

Example: Performing vulnerability analysis of an artifact

Suppose that an artifact version is used in various application models and master components and that the artifact is deployed to various environments. Also suppose that it is discovered that the particular artifact version has a security vulnerability. In such cases, it might be important to quickly know the usage of that artifact version so that corrective action can be taken.

Example: Understanding dependencies

Suppose that users have created application components using a copy of a master component. If the definition of that master component changes, you should understand the dependency and impact so that owners of applications that leverage the master component can take corrective action.

You can use the Get Artifact Usage procedure in the CloudBees project to understand how and where an artifact and component is used to perform vulnerability analysis or to understand dependencies or any other artifact usage use case. When this procedure is run, it asks for inputs such as a component artifact name and version as well as an email address for sending out a usage report.

Figure 18. Get Artifact Usage procedure

For example, suppose that you want to generate a report for the usage of an artifact named Shared:Login with version 4.2 and then email the report to a user. You can just enter that information and run the procedure. This generates a report that shows clearly which master components, applications, and environments have links to the 4.2 version of the Shared:Login artifact.

Figure 19. Generate report

Users can easily understand the current usage of an artifact or a master component, and the system emails this report to the respective users. This procedure can be extended easily to take corrective actions if needed, such as redeploying a new version of the artifact to all the current environments to remove the security vulnerability.

Understanding the artifact cache

Artifact versions are retrieved to an artifact cache directory before being consumed by a build process. Properly configured, the artifact cache directory can significantly improve efficiency as you work with artifacts.

The artifact cache directory is set in one of these ways:

Enter information in the Artifact Cache Directory field on the Edit Resource page.
Use the modifyResource API to set the artifactCacheDirectory intrinsic property.
Set the artifactCache field in the agent.conf file.

Thus, each resource can have its own artifact cache location (directory). Alternatively, multiple resources and an entire site can share an artifact cache if the cache directory resides on a shared file server.

When a CloudBees CD/RO step retrieves an artifact version, remember that often a version range or no version at all is specified. For example, to retrieve the latest CloudBees CD/RO SDK artifact version, you could issue the following command:

ectool retrieveArtifactVersions --artifactName commander:SDK▼

No version is specified because we want the "latest" artifact version published.

In the following example, the step sends a request (1) to the CloudBees CD/RO server:

The CloudBees CD/RO server replies (2) with a fully qualified artifact version name. In this case, it returns commander:SDK:1.2.0.43552.
The step then looks in its artifact cache for the artifact version (3).
If the artifact is not found in the artifact cache, a request is issued (4) to get a list of repositories to search to find the artifact version.
The CloudBees CD/RO server replies (5) with a list of repositories and the order in which to search them.
The step requests the artifact version from the first (6) repository in the list.
If the artifact version is found in the repository, the artifact version is retrieved to the cache (7).
If the artifact version is not found, the step requests the artifact version from the next repository in the search order (8).
If the artifact version is found in that repository, the artifact version is retrieved to the cache (9).
Otherwise, if the artifact version is not found in any repository, an error is displayed.

Figure 20. Artifact cache

In this example, if the requested artifact version was found in the artifact cache (step 3), the resource would not have communicated with either of the repository servers. Also, any potential network latencies would have been avoided from retrieving the artifact version from the repository server.

If you have a cache shared by multiple resources and steps that run on those resources need to use the same artifact version, the first step requesting an artifact version retrieves it from the repository to the cache. Subsequent job steps using the artifact version have immediate access to the artifact version in the cache, eliminating the need for each step to retrieve its own copy of an artifact version.

Cleaning up repositories and caches

This method does not work for artifacts in an Amazon Simple Storage Service (Amazon S3) repository.

Deleting an artifact version in the CloudBees CD/RO server does not delete it in the repository backing store, and it does not remove the artifact version from an artifact cache.

ectool and the CloudBees CD/RO Perl module include a function for each of these tasks: cleanupRepository and cleanupArtifactCache. Both of these API calls are intended to be run on the machine containing the cache/backingstore directory that needs to be cleaned up.

The cleanup algorithm works as follows:

Reviews the directory tree looking for directories three-levels deep.
For each such directory, interprets the three path components as < groupId, artifactKey, version >.
Issues a findObjects request to the CloudBees CD/RO server to check whether an artifact version with that specification exists.
Deletes all directories identified by < groupId, artifactKey, version > and not recognized by the CloudBees CD/RO server.

To operate correctly, using the cleanupRepository API requires read access to all artifact version objects stored in CloudBees CD/RO and therefore must be run in an "admin" session. CloudBees also recommends running cleanupArtifactCache in an "admin" session—if not, artifact versions that are simply not readable by the user invoking the cleanup are deleted from the cache.

In addition, both functions fail if the server does not recognize any of the chosen artifact versions. Thus, if the cleanup function is called on a directory that is not a backingstore directory or and artifact cache, it does not delete anything.

The recommended approach for removing stale artifact versions from a backing store is to use a CloudBees CD/RO maintenance procedure that periodically runs on each repository server (which presumably has an agent install). Similarly, use a procedure with a broadcast step to run on each agent to clean up its cache.

The job step session may not have read privileges on all artifact versions. CloudBees recommends that you sign in as a privileged user, and then perform cleanup with those credentials.

Artifact management overview

About artifact objects

Artifact

Examples

Artifact version

Version strings

The artifactVersionState property

Artifact repository

Using multiple repositories

Copying artifacts to multiple repositories

Entering parameters

Understanding overwrite behavior

Troubleshooting

Configuring the artifact repository server for Amazon S3

Configuring the artifact repository server for Google Cloud Platform

The respositoryDisabled Intrinsic Property

Configuring access control

Control who can create new artifacts and publish artifact versions

Control which projects can retrieve artifact versions at run time

Group 1 publishes an artifact version and wants only Group 2 to be able to retrieve it

Uploading and publishing artifacts from a local file system using the UI

Uploading files to publish a new version of an existing artifact

Creating a New Artifact and Uploading Files to Publish the First Version

Publishing artifact versions

Enabling compression

Using include and exclude patterns

Retrieving Artifact Versions

Parameters and environment variables for artifact retrieval retry attempts

Entering filters during a retrieve operation

Entering dependent artifact versions

Understanding artifact usage

Example: Performing vulnerability analysis of an artifact

Example: Understanding dependencies

Understanding the artifact cache

Cleaning up repositories and caches

The `artifactVersionState` property

The `respositoryDisabled` Intrinsic Property