FileOps plugin

16 minute readExtensibilityDeveloper productivity

The EC-FileOps plugin is open, it allows the input of any data defined in the provided fields of the parameters. Is important to provide all the parameters required for the correct functioning of the FileOps plugin.

CloudBees CD/RO itself provides some advantages like parameter checking, easy functionality selection using only check boxes or radio buttons, historical log reports, procedure duration statistics, simple credentials attachment, real time analysis of the log been generated, among others.

This plugin provides CloudBees CD/RO procedures you can use to perform the following operations:

  • Add text to file

  • Change ownership or permissions

  • Check file permissions

  • Compare two files checksums

  • Move or copy files and directories "to" or "from" a CloudBees CD/RO resource.

  • Create a new directory.

  • Create an empty file.

  • Create a symbolic link.

  • Create a zip file.

  • Delete a specific directory.

  • Delete a particular file.

  • Describe file.

  • Copy files or directories between CloudBees CD/RO resources, using a CloudBees CD/RO-native protocol.

The Remote Copy-Native procedure:

  • No set up required.

  • Only works with CloudBees CD/RO resources for source and destination specifications.

  • Files are uploaded form the source resource to the CloudBees CD/RO server, then downloaded to the destination resource.

  • Copy files or directories between an CloudBees CD/RO resource and an arbitrary host (including another CloudBees CD/RO host) or between CloudBees CD/RO resources, using SCP.

The Remote Copy-SCP procedure:

  • Requires an SSH server setup on one end-point and a defined credential to verify authentication.

  • Requires only one end-point to be a resource.

  • Files are copied directly from the source to the destination host.

Plugin Version 3.0.1.2022040516 Revised on April 13, 2022

Integrated version

This plugin was developed and tested against FileOps version 3.0.1.2022040516

Updated Perl version required

This plugin is using an updated version of Perl, cb-perl shell (instead of old ec-perl). This means that it needs CloudBees CD agent version 10.3+ in order to work.

Plugin procedures

In the CloudBees CD UI, from the Home page, open the Main Menu, and select Admin > Plugins to open the Plugin Manager.
For all parameter descriptions below, required parameters are shown in bold italics.

Add text to file

This procedure adds text to a file.

Parameter Description

Path

Path to the file you want to write in.(Required)

Append Mode

If checked, it will append new text to the file rather than replace it.(Required)

Add new line to the end

If checked, it will add a new line at the end of the file.(Required)

Content

Content to write in the file.(Required)
image

Change ownership or permissions

Changes the Ownership and/or Permissions from files and folders.

NOTE: This procedure is not implemented for Windows OS.

Parameter Description

Path

Path to the file/directory.(Required)

Owner

Name of a valid user.

Mode

New mode to apply, such as 0666 (read and write permission for everyone).

Recursive

If checked, Applies mode/owner recursively to directories.(Required)

Verbose Mode

If checked, displays additional information about the process.(Required)

Check file permissions

Checks if Specific User has Permissions on a File.

NOTE: This procedure is not implemented for Windows OS.

Parameter Description

UserName

Name of the user to validate. (Required)

Path

Path to the file to validate. (Required)

Read

If checked, Validates if user is allowed to read the file.

Write

If checked, Validates if user is allowed to write in the file.

Execute

If checked, Validates if user is allowed to execute the file.

Compare 2 files checksums

Compare the checksum of 2 files.

Parameter Description

First File

Path to the first file to compare.(Required)

Second File

Path to the second file to compare.(Required)

Copy file

Copy files and directories "to" or "from" an CloudBees CD resource.

Parameter Description

Source:

Directory or file to copy. This may end in a glob pattern to copy a subset of files to the destination.(Required)

Destination:

Where to copy source file/directory to.(Required)

Replace destination if it exists:

If set, and if the destination is a pre-existing directory, it is recursively cleared prior to copying source into it. If not set, and if the destination is a pre-existing file, the step errors out.

Create directory

Creates a new directory.

Parameter Description

Path

Path of the directory to create.(Required)

Create empty file

Creates a new empty file.

Parameter Description

Name

Path of the file to create.(Required)

Permission Mode

Permission to apply after creating the file, such as 0666 (read and write permission for everyone).

Creates a symbolic link between target and destination.

Parameter Description

Target

Target to link. (Required)

Destination

Place where the symlink will be located. (Required)

Create zip file

Create a zip file from a directory or file.

Parameter Description

Source Path:

File or directory to zip. This may end in a glob pattern to zip a subset of files in a directory. (Required)

Zip File:

Name of zip file in which to store source file(s). The zip file must not exist apriori. (Required)

Delete directory

Deletes a directory.

Parameter Description

Path

Path to the directory to delete. (Required)

Recursive

If checked, it deletes a directory and all its content. (Required)

Delete file

Deletes a given file.

Parameter Description

Path

Path to the file to delete. (Required)

Describe file

Prints basic and technical information about a file.

Parameter Description

Path

Path to the file to describe. (Required)

Diff files

Shows differences among files in different formats.

Parameter Description

First File

Path to the first file to diff. (Required)

Second File

Path to the second file to diff. (Required)

Display Format

Choose the display format when showing results. Options are: Table, Unified and Context. (Required)

Diff folders

Shows differences among Folders.

Parameter Description

First Directory

Path to the first folder to diff. (Required)

Second Directory

Path to the second folder to diff. (Required)

Suppress Identical File Messages

If checked, Suppress messages about identical files. (Required)

Directory exists

Verify if a directory exists or not.

Parameter Description

Path

Path of the directory to check. (Required)

Procedure Results Handling Strategy

The strategy of procedure results handling. "Default" behavior is to fail when the directory does not exist. When the strategy is "Property," the procedure will not fail, but appropriate output parameters/properties will be set.

Output PropertySheet Path

A PropertySheet to be created after the procedure is finished. The procedure will create two properties at the path: "Exists" and "Path". Exists property is boolean, will be 'true' or 'false'. The "Path" property will contain the entity path from the "Path" parameter.

Crypt

Encrypts and Decrypts a file using the Blowfish algorithm

Parameter Description

File

Path to the file to Encrypt or Decrypt. (Required)

Action

Encrypt or Decrypt.

Key

A secret key.

BackUp file

If checked, creates a copy of the file before encrypting/decryptingit.

File exists

Verify if a file exists or not.

Parameter Description

Path

Path of the file to check. (Required)

Procedure Results Handling Strategy

The strategy of procedure results handling. "Default" behavior is to fail when the file does not exist. When the strategy is "Property," the procedure will not fail, but appropriate output parameters/properties will be set.

Output PropertySheet Path

A PropertySheet to be created after the procedure is finished. The procedure will create two properties at the path: "Exists" and "Path". Exists property is boolean, will be 'true' or 'false'. The "Path" property will contain the entity path from the "Path" parameter.

Generate checksum

Generate an MD5 or SHA1 checksum for the given file.

Parameter Description

File:

File whose checksum is to be computed. (Required)

Checksum Type:

SHA-1 or MD5. (Required)

Result Property:

Name of property to store result of checksum computation. (Required)

List files in folder

Lists all files in a folder.

Parameter Description

Target Directory

Place where the procedure will look for files to list. (Required)

Filter

Filter to apply when listing files. (for example, *.txt).

Separator

A custom separator for files in the folder.

Move file

Moves a file.

Parameter Description

Source:

Directory or file to move. This may end in a glob pattern to move a subset of files to the destination. (Required)

Destination:

Where to move source file/directory to. (Required)

Replace destination if it exists:

If set, and if the destination is a pre-existing directory, it is recursively cleared prior to moving source into it. If not set, and if the destination is a pre-existing file, the step errors out.

Remote Copy File (Native)

Copy a file/directory from one CloudBees CD resource to another, using CloudBees CD’s built-in getfiles/putfiles mechanism.

Make sure the Stomp port, by default 61613, is opened on the CloudBees CD server. This port must be opened bi-directionally if you want to transfer files both to and from the server. Details: /d/kb-360033190051[KBEC-00352 - Configuring Stomp for Preflight and EC-FileOps file transfers].
Parameter Description

Source Resource

Resource to copy file/directory from. (Required)

Source Workspace

Workspace on the source resource containing the file/directory to transfer. If a file/directory outside of the workspace is to be copied, this workspace is still used for storing temporary files. In that case, it is recommended that this workspace refer to a local disk directory on the resource.

Source Path

File or directory to copy. This may end in a glob pattern to copy a subset of files to the destination. (Required)

Destination Resource

Resource to copy file/directory to. (Required)

Destination Workspace

Workspace on the destination resource into which the source file/directory is copied. If the destination location is outside of the workspace, this workspace is still used for storing temporary files. In that case, it is recommended that this workspace refer to a local disk directory on the resource.

Destination Path

Location on destination resource to copy source file to. (Required)
remote copy file native
Source Path Destination Path Behavior

/var/tmp/simple.txt

simpleNew.txt

simpleNew.txt is created in the destination workspace from the contents of /var/tmp/simple.txt.

/var/tmp/simple.txt

preexistFile.txt

preexistFile.txt contents are replaced with those of /var/tmp/simple.txt.

/var/tmp

preexistDir

preexistDir contains a 'tmp' subdirectory with the contents of /var/tmp. If preexistDir was originally non-empty, those directory elements remain.

/var/tmp

newDir

newDir is created and contains the contents of /var/tmp.

/var/tmp/*

newDir2

newDir2 is created and contains the contents of /var/tmp.

/var/tmp/s*

newDir2

newDir2 is created and contains a copy of each file or directory in /var/tmp that begins with "s".

Rename file

Renames a file.

Parameter Description

Old Name

Path to the file you want to rename. (Required)

New Name

Provide a new name for the file. (Required)

Save file content in property

Read a file content and store its content in a property.

Parameter Description

Path

Path to the file to read. (Required)

Content (output property path)

Property name used to store the file content. (Required)

Sync folders

Perform folder diff to two folders and sync their content.

Parameter Description

First Directory

First folder to sync. (Required)

Second Directory

Second folder to sync. (Required)

Sync Rule

Rule to apply when synchronizing folders. Options are: Keep First, Keep Second and Mirror Mode. (Required)

Dry Run

If checked, it will print the copy/delete operations but will never perform the real work.

Verbose Mode

If checked, it will print additional information about the sync process.

Actions

For Sync Rules 'Keep First' and 'Keep Second':

A:Source B:Dest Type Action

exists

not exists

File

copy A → B

exists

not exists

Directory

mkdir B

not exists

exists

File

unlink B

not exists

exists

Directory

rmdir B

newer

older

Directory

copy A → B

older

newer

File

copy B → A

NOTE: for 'Keep Second' rule source and destinations are swapped.

For Sync Rule 'Mirror':

A:Source B:Dest Type Action

exists

not exists

File

copy A → B

exists

not exists

Directory

mkdir B

not exists

exists

File

no action

not exists

exists

Directory

no action

newer

older

Directory

copy A → B

older

newer

File

no action

Truncate file

Truncates a file to a specified length.

Parameter Description

Path

Path to the file to truncate. (Required)

Length

Adjust the file size, in bytes. (default '0')(Required)

Unzip file

Unzip a zip file to a directory.

Parameter Description

Zip File:

Zip file to extract. (Required)

Destination Directory:

Directory to extract into. Need not exist apriori. (Required)

Update file last touch

Update last accessed and modified date in a file.

Parameter Description

Path

Path to the file to "touch". (Required)

Year

The year part of the date to set in the file. (Required)

Month

The month part of the date to set in the file. (Required)

Day

The day part of the date to set in the file. (Required)

Hour

The hour part of the date to set in the file.

Minute

The minute part of the date to set in the file.

Second

The second part of the date to set in the file.

TimeZone

A valid timezone for the date, such as "America/Costa_Rica", "America/Chicago". (Required)

Validate checksum

Validate that the given file has the given MD5 or SHA1 checksum.

Parameter Description

File

File whose checksum is to be validated. (Required)

Checksum Type

SHA-1 or MD5. (Required)

Expected Checksum

Expected checksum of the file. (Required)

Remote Copy File (SCP) Step

This step copies a file or directory between an CloudBees CD resource and some other machine (possibly a different CloudBees CD agent) using the SCP protocol. Unlike other copy steps, this remote copy step does not generally create intervening directories in the destination path if needed.

Note: If the destination is a pre-existing file, it will be overwritten. If the destination is a pre-existing directory, pre-existing files/directories in it will be overwritten if there are corresponding elements in the source.

Copy configurations

There are three copy configurations available for this step. Each configuration supplies its own set of fields for you to supply the necessary information for the step to operate properly.

CloudBees CD resource ⇒ Non-CloudBees CD host

In this configuration, the SCP operation is driven by a step on an CloudBees CD resource that establishes an SSH connection with an arbitrary host and uploads files or directories to that host.

Parameter Description

Source Resource

Name of the resource from which file(s) will be copied. The resource initiating the SSH connection and performing the upload. (Required)

Source Workspace

Name of workspace on source resource from which file(s) are to be copied. (Required)

Source Path

Path of source file/directory to copy. If relative, this path is evaluated relative to the source workspace.

Relative or absolute path for the files or directories to upload. A relative path is processed relative to the Source Workspace. This path may end in a glob pattern to copy multiple files from a source directory to the destination. (Required)

Destination Host

Destination Host Name/IP of the host to which file(s) will be copied. It must run an SSH2 server listening on port 22. (Required)

Credential

Property path to credential for authenticating with the SSH server running on the destination machine.

Note: This field cannot be set during step creation, but it is required. You must save the step and then edit the step to set the credential parameter. (Required)

Destination Path

Path of destination file/directory to copy. (Required)

Debug Level

Level of debug logging to emit for the step. (optional)
remote copy file scp

Non-CloudBees CD host ⇒ CloudBees CD resource

In this configuration, SCP operation is driven by a step on an CloudBees CD resource that establishes an SSH connection with an arbitrary host and downloads files or directories from that host.

Parameter Description

Source Host

Name/IP of the host from which file(s) will be copied. This host must be running an SSH2 server listening on port 22. (Required)

Credential

Property path to credential for authenticating with the SSH server running on the source machine.

Note: This field cannot be set during step creation, but it is required. You must save the step and then edit the step to set the credential parameter. (Required)

Source Path

Path of source file/directory to copy.

This path may end in a glob pattern to copy multiple files from a source directory to the destination. (Required)

Destination Resource

Name of the resource where the file(s) will be copied.

The resource that initiates the SSH connection and performs the download. (Required)

Destination Workspace

Name of workspace on destination resource to which file(s) will be copied. (Required)

Destination Path

Path of destination file/directory to copy.

Relative or absolute path for the files or directories to where they will be downloaded. A relative path is processed relative to the Destination Workspace. (Required)

Debug Level

Level of debug logging to emit for the step.(optional)

CloudBees CD resource ⇒ CloudBees CD resource

In this configuration, the SCP operation is driven by a step on an CloudBees CD resource that establishes an SSH connection with another CloudBees CD resource and either downloads or uploads files or directories from or to that resource. This configuration is particularly useful for copying files between different workspaces through the resources that can access those workspaces.

When this configuration is chosen, a few additional fields are presented:

  • End-point that has an SSH server - Indicates whether the source resource or destination resource is used as the SSH server. This in turn sets which resource runs the step and which resource requires the credential (indicated by the location of the Credential field in the form).

  • SSH server machine is a Windows host - Used by the step to convert a potentially relative path on the SSH server to absolute, using the appropriate workspace attribute -agentUncPath for Windows or agentUnixPath for non-Windows machines.

The lower Parameter section contains the following form fields:

Parameter Description

Source Resource:

Name of the resource from which file(s) will be copied.

The resource that contains the source files or directories. (Required)

Source Workspace:

Name of workspace on source resource from which file(s) are to be copied. (if the destination resource is the SSH server) and the base directory for computing the absolute path to the source files or directories if Source Path is relative. (Required)

Source Path:

Path of source file/directory to copy. If relative, this path is evaluated relative to the source workspace.

This path may end in a glob pattern to copy multiple files from a source directory to the destination. (Required)

Destination Resource:

Name of the resource to which file(s) will be copied. (Required)

Destination Workspace:

Name of workspace on destination resource to which file(s) are to be copied. (Required)

Destination Path:

Path of destination file/directory to copy. If relative, this path is evaluated relative to the destination workspace. (Required)

Debug Level:

Level of debug logging to emit for the step. (optional)

Credential

Property path to credential for authenticating with the SSH server running on the destination machine.

Note: This field cannot be set during step creation, but it is required. You must save the step and then edit the step to set the credential parameter. This field is located after Source Resource if the source resource is the SSH server, otherwise it is located after Destination Resource.

Copying as the destination path versus into the destination path

As stated earlier, the source file may end in a glob pattern if copying multiple files from a source directory to the destination. Depending on whether or not the destination exists, the source may be copied as the destination or into the destination.

See the following table for examples illustrating step behavior in various circumstances (assuming resource to resource for simplicity, though these examples also apply for other configurations with absolute paths instead of relative paths):

Source Path Destination Path Behavior

/var/tmp/simple.txt

simpleNew.txt

simpleNew.txt is created in the destination workspace with /var/tmp/simple.txt contents.

/var/tmp/simple.txt

preexistFile.txt

preexistFile.txt contents are replaced with the contents from /var/tmp/simple.txt.

/var/tmp

preexistDir

preexistDir contains a 'tmp' subdirectory containing the contents of /var/tmp. If preexistDir was originally non-empty, those directory elements remain.

/var/tmp

newDir

newDir is created and contains the contents of /var/tmp.

/var/tmp/*

newDir2

newDir2 is created and contains the contents of /var/tmp.

/var/tmp/s*

newDir2

newDir2 is created and contains a copy of each file or directory in /var/tmp that begins with "s".

Attaching credentials

All cases for this procedure require credentials, however, the credential field cannot be set during step creation, but it is required. It is necessary to save the step and then edit the step to set the credential parameters. To make this more clear an example is provided:

The first thing to do is create a credential in the project that is being used, in order to do that just go to the "Credentials" tab and select "Create Credential" then provide the parameters:

It is possible to check the created credentials:

After that the next thing to do is set the parameters required normally:

At this point the step is created, but still does not have a credential attached, so next we enter the step again:

Check below for the "Attached Credentials" option and select "Attach Credential":

By doing this it is possible to attach the credential earlier created in the project:

After that is ready, it is necessary to provide the credential parameter in order to run the step, so we go up and enter "Browse" on the Credential parameter:

and finally select the credential on the current project:

Following this steps it is possible to execute the Remote Copy using SCP procedure.

Use cases and examples

Example 1: common file operations

In this example we are going to perform several operations in a specific directory.

Procedure CloudBees CD:

  1. Create a directory.

  2. Create 2 files in it.

  3. Create a Zip file from the directory.

  4. Delete the directory

  5. and finally Unzip the zip file.

First of all, we are going to create a project called "Use Cases"

Create a procedure called "Use Case 1"

Create a new parameter called "basedir" in the procedure we just created.

Create a EC-FileOps’s step from the step picker called "Create Directory" and fill it as seen here:

Create a EC-FileOps’s step from the step picker called "Create Empty File" and fill it as seen here:

Create a copy of the previous step but this time fill it with a different file name:

Create a EC-FileOps’s step from the step picker called "Create Zip File" and fill it as seen here:

Create a EC-FileOps’s step from the step picker called "Delete Directory" and fill it as seen here to erase the directory we created before:

Finally Create a EC-FileOps’s step from the step picker called "Unzip File" to unzip the file we created in the "Create Zip File" step:

Now our procedure should look like:

The procedure is finally ready to be executed. Fill the "basedir" with a directory path, in this case I used the /tmp directory but you can use whatever you want.

After running the procedure you should be able to see something similar to the next screenshot:

The procedure we created is 100% portable, so you can run it on a windows machine just by changing the "basedir" parameter to a windows path such "D:/".

Example 2: safely download a file from the internet

In this example we will create a procedure to download a file from internet and compare its checksum.

Procedure CloudBees CD:

  1. Download a file from the Internet. (Ant)

  2. Download the file’s md5 checksum.

  3. Compare the file with the checksum

Create a procedure called "Use Case 2" in the project "Use Cases"

Create a new parameter called "basedir" in the procedure as we did in the first use case.

Create a Command step that we will use to download ant using "wget"

Create a EC-FileOps’s step from the step picker called "Save File Content in Property" and name it "Read Checksum" and fill the parameters:

Finally it’s time to compare the file checksum by creating a new step called "Validate Checksum" from the EC-FileOps’s available steps:

At this point our procedure should look like this:

Run the procedure and fill the basedir parameter with a base path. In this case I used the /tmp folder.

After running the procedure we should see something like:

Job properties:

Example 3: diff two different files

In this example we will create a procedure to show differences among two files.

Procedure CloudBees CD:

  1. Create two different files

  2. Compare their content and create a nice log showing differences among them.

Create a procedure called "Use Case 3" in the project "Use Cases"

Create a new parameter called "basedir" in the procedure as we did in previous examples.

Create a EC-FileOps’s step from the step picker called "Add Text To File" and fill it as seen here:

Create a copy of the previous step and change the parameters Path and Content as seen here:

Create a EC-FileOps’s step from the step picker called "Diff Files" and fill it as seen here:

At this point our procedure should look like this:

Run the procedure with the basedir parameter "/tmp"

Expected log:

Example 4: encrypt and decrypt files with CloudBees CD

In this example we will encrypt and decrypt a file

Procedure CloudBees CD:

  1. Choose the file to encrypt and decrypt

  2. Encrypt the file

  3. Decrypt the file

Create a file and name it "file.txt" and store it in the /tmp folder

From the step picker, choose the step called "Encrypt-Decrypt File" and fill the parameters:

Run the procedure and open the log file in the browser:

To make sure the file was encrypted I am going to use "cat" to print the file in console:

As we could see, the file is encrypted now. So it’s time to create our Decrypt procedure.

From the step picker, choose the step called "Encrypt-Decrypt File" and fill the parameters:

Run the procedure and open the log file in the browser:

To make sure the file was decrypted I am going to use "cat" to print the file in console:

Example 5: sync two folders

In this example we will synchronize the content of two folders.

Let’s synchronize two folders by creating a Sync Folders step from the step pickers:

When you run the procedure, the step will create a log with the actions that were performed.

Alternatively, you can check the "Dry Run" parameter to simulate the actions that would be executed in a real scenario.

Dry Run log:

Release notes

EC-FileOps 3.0.1

  • Upgraded HttpClient library to v4.5.13.

EC-FileOps 3.0.0

  • Upgraded from perl-5.8 to perl-5.32. No backward compatibility with releases prior 10.3. Starting from this release new agent is required to run this plugin’s procedures.

EC-FileOps 2.1.1

Fixed a bug when the "Move" has been cleaning up destination folder if source is a file.

EC-FileOps 2.1.0

  • FileExists and DirectoryExists procedures have been improved. Now one can control how the procedure behaves, and procedure sets output properties/output parameters.

EC-FileOps 2.0.10

  • The issues with procedure "Remote Copy - Native" in pipeline context have been fixed.

EC-FileOps 2.0.9

  • The documentation has been migrated to the main documentation site.

EC-FileOps 2.0.8

  • Renaming to "CloudBees CD"

EC-FileOps 2.0.7

Renaming to "CloudBees"

EC-FileOps 2.0.5

  • Move agent code to plugin’s properties.

  • Fixed Copy/Move procedures for paths that contains spaces.

  • Fixed ListFiles procedure for paths that contains spaces.

  • Validate datetime parameter in UpdateLastTouch procedure.

  • Fixed scp copying for path that contains spaces.

  • Updated documentation for Sync Folders procedure, clarify actions for rules.

  • Fixed: Remote Copy - SCP fails on linux with "ec-perl: Command not found" error.

  • Fixed Encrypt/Decrypt procedures under Windows.

EC-FileOps 2.0.4

  • Renamed ElectricCommander to ElectricFlow.

EC-FileOps 2.0.3

  • Fix documentation of "Remote Copy File (SCP) step".

EC-FileOps 2.0.2

  • Fix link to help page.

  • Fix create zip file with space in path.

EC-FileOps 2.0.1

  • Several bugs were fixed.

EC-FileOps 2.0.0

  • 22 brand new procedures were added.

  • Help page was upgraded to the new format.

  • Use cases were added to the help page.

FileSysRepo plugin