Preflight Builds

Why use Preflight Builds?

When developers make changes, they generally build and test their code locally and commit their changes if the code ran successfully. Unfortunately, this process is not sufficient.

Code changes may break the production build because of environment differences, platform-specific issues, or incomplete commits. These issues leave the product in a broken state until the changes are backed-out of the product or the developer commits a fix, which often affects the entire development team. It is a time-consuming process to find the problem when multiple developers commit their changes simultaneously and the build breaks.

Preflight Build Solution

A Preflight Build is used to build and test a developer’s changes before those changes are committed. A "post-commit" source tree is simulated by creating a clean source snapshot and overlaying the developer’s changes on top of it. These sources are then passed through the production build procedure to validate the changes work successfully.

Developers are allowed to commit their changes only if the preflight build is successful. Because developer changes are built and tested in isolation, many common reasons for broken production builds are eliminated.

Preflights with CloudBees Flow

Workflow

After a preflight build is configured, the general flow of interaction between the server, agent, and client is:

  • A developer invokes the client preflight program either through an IDE or via the command-line. The following arguments are specified in the XML configuration file or on the command-line:

  • Define which CloudBees Flow server to use (hostname, ports, and so on),

  • Define the project, procedure, and parameters to use to start the preflight build, and

  • SCM-specific information to use (ports, changelists, and so on).

  • Next, the client preflight program connects to the CloudBees Flow server and launches the specified procedure.

  • The job is started and the developer’s changes are uploaded.

  • If auto-commit was turned on, the client program waits for the job to complete. Otherwise, the client program exits immediately.

  • The job progresses until it gets to the special step added to the procedure to extract sources and overlay changes.

  • The agent-side driver creates a clean source snapshot based on configuration information passed to it from the client-side.

  • The step overlays changes [uploaded by the client] on top of the snapshot to simulate the developer’s check-in.

  • The build continues as usual, using the modified sources.

  • If the developer chose to auto-commit changes, the preflight program was waiting for the job to complete. If the job is successful, the changes are committed if:

  • No files were added or removed from the change sets.

  • No files were modified since the preflight build started.

  • All conditions set in the driver script were met.

Components

Three components are required for a CloudBees Flow preflight build:

  • Server —This is the main CloudBees Flow server that will run the preflight builds.

  • Agent —This is any CloudBees Flow agent machine that is setup as a resource for the server. This machine must be able to communicate with the source control system because it is responsible for creating a clean source snapshot on which the developer’s changes are overlayed.

  • Client —This is the machine where a developer has active code changes to submit for a preflight build. Like the agent, this machine must be able to communicate with the source control system because it is responsible for determining which files were modified so it can transmit these files to the agent.

Installation

Preflight is automatically installed with SCM plugins. CloudBees Flow pre-installs numerous source code management (SCM) plugins including Accurev, Bazaar, CVS, ClearCase, Git, Mercurial, Perforce, SVN, StarTeam, Team Foundation Server, and Vault. Before you can use your preferred SCM, you will need it to communicate with the CloudBees Flow server.

Configuration

This section provides details for setting up and running Preflight builds in a typical build environment.

Server

Adding the preflight snapshot step

A typical production build procedure has a step that extracts a clean source code snapshot before the actual build starts. For preflight builds, a special step needs to run in place of this snapshot step. This special step is responsible for creating the base snapshot and then overlaying the developer’s changes.

  • On the Procedure Details page, to create a new step click the Plugin link.

  • On the Choose Step panel, select Source Code Management.

  • Choose the ECSCM plugin for your SCM or from the right-pane, select the "Preflight" step for your SCM.

  • On the New Step page, notice that your SCM is displayed in the Subprocedure section.

  • Set the Resource name for the step to an agent that can communicate with the SCM system, so it can create the source snapshot.

  • The Parameter section also displays appropriate fields to enter information for your SCM. The following two fields are common to most SCMs:

  • Configuration–Enter the name you created for your SCM configuration.

  • Destination Directory–This is a path relative to the job’s workspace, where the source tree will be created.

  • Enter all other information required for your new step.

  • Click OK.

Choosing between snapshot steps

Only one snapshot step should run depending on whether a production or preflight build is invoked. To enable the appropriate step, add a checkbox-style parameter to the build procedure to determine whether or not the build is a preflight. Now, use that parameter in the Run Condition field on the two snapshot steps. In the following example, the parameter is named "preflight", with an unchecked value of "false" and a checked value of "true".

  • For the preflight snapshot step, set the run condition to $[preflight]

  • For the production snapshot step, set the run condition to $[/javascript getProperty("preflight") == "false"]

Choosing how files are uploaded

By default, when developers run preflight builds, their changes are uploaded to the job workspace via the server. However, if the job workspace is on a network share, accessible to the developer’s machine, a useful optimization is for files to be copied directly into the job workspace. To use the network share approach, set a flag on the top-level procedure that is invoked when running a preflight build. This property can be created on the Procedure Details web page as follows:

  1. Create a nested property sheet on the procedure called ec_preflight.

  2. Create a property in that procedure called waitForStep with a value of "1". This property can be created from the command-line by calling: ectool setProperty ec_preflight/waitForStep 1 --projectName <project> --procedureName <procedure>

Agent

The agent machine must be able to communicate with the SCM system to create the source snapshot.

Client

From a developer's machine, there are two ways to start preflight builds:
  • From an IDE:

  • Eclipse Open a Run dialogue, select Launch Commander Procedure, then select an existing preflight configuration or create a new one. For more information, see the CloudBees Flow Automation Platform Eclipse Integration Tech Note PDF file or the Eclipse plugin.

  • Visual Studio (See the CloudBees Flow Automation Platform Visual Studio Tech Note PDF file or the Visual Studio plugin)

  • From the command line:

    The executable program, ecclientpreflight, is used to run a preflight build from the command-line. This program is included in the CloudBees Flow "Tools" installation. When you run ecclientpreflight, configuration options can be passed in an XML configuration file or on the command-line.

    If the same option is specified in both the configuration file and on the command-line, the value passed on the command-line takes precedence. The XML configuration file can be passed to the program via the --config option. If the XML configuration file is not passed in, current and ancestor directories are searched for a file named " .preflight ", using the first one it finds as the configuration file.

    Generally, a " .preflight " file is stored at the root of a developer’s workspace, containing configuration information specific to that workspace.

Samples

In the following samples, the SCM was set to Perforce. Some options may be specified using an abbreviated format ( -c, with a single dash) or a more verbose format ( --config, with two dashes).

General Options

-c,--config <file>

Load configuration options from the specified XML file. If <file> is '-', read standard input. If not supplied, the current and ancestor directories will be searched for a .preflight file. The first .preflight found will be used as the preflight configuration.

--load <file>

Evaluate <file> after option processing. May appear multiple times.

-d,--debug

Display debugging information.

-v,--version

Display version information.

-h,--help

Display this information.

Server Communication Options

--server <name>

The host name (defaults to localhost).

--port <port>

The HTTP port (defaults to 8000).

--securePort <port>

The HTTPS port (defaults to 8443).

--secure

If set, only secure connections are used. Off, by default.

--timeout <timeout>

The response timeout in seconds (defaults to 180).

--userName <name>

The name of the CloudBees Flow user to login as.

--password <password>

The password for the specified user. If blank and the user has no active session, a prompt to enter the password will be provided.

--driverLocation

Property path to a property sheet containing the driver scripts that will be loaded and run after a connection to the server is established. Defaults to /server/ec_preflight.

--mainDriver

The name of a property in the sheet specified by driverLocation. This is the main driver script that will be loaded and run after a connection to the server is established.

Logging Options

-l,--log

If specified, debug information is logged. Off, by default.

--logDir

Where to store log and other files. Defaults to the user’s home directory.

Procedure Invocation Options

--projectName <name>

The name of the CloudBees Flow project containing the procedure to invoke.

--procedureName <name>

The name of the CloudBees Flow procedure to invoke.

-p,--param <name>=<value>

Enter additional parameters to the procedure. May appear multiple times. If any parameters were specified in the config file, those supplied on the command-line append to that list, overriding parameters with the same name.

--priority

The priority of the job. Possible values are low, normal, high, highest. If left unspecified, defaults to normal.

--jobTimeout <timeout>

The number of seconds to wait for the job to complete when auto-committing changes. Defaults to 3600 seconds (1 hour).

--waitForJob

Wait for the job to complete and report its outcome. This action does not occur by default unless a set of SCM charges are being committed automatically.

`--runOnly `

Run the procedure and exit immediately. The SCM driver is not downloaded in this case.

SCM Options

--scmType

The name of the SCM, required, unless the procedure is invoked in 'run only' mode. The driver is downloaded from $driverLocation/clientDrivers/$scmType.

Valid values: ECSCM-Accurev, ECSCM-Bazaar, ECSCM-ClearCase, ECSCM-CVS, ECSCM-Git, ECSCM-Mercurial, ECSCM-Perforce, ECSCM-Repo, ECSCM-StarTeam, ECSCM-SVN, ECSCM-TFS, ECSCM-Vault.

--autoCommit <1|0>

Whether or not the changes should be automatically committed if the job completes successfully. Off, by default.

--commitComment <comment>

A comment for the auto-commit.

Perforce Options

--p4port <port>

The value of P4PORT. May also be set in the environment. This is a required value.

--p4user <user>

The value of P4USER. May also be set in the environment. This is a required value.

--p4passwd <password>

The value of P4PASSWD. May also be set in the environment.

--p4client <client>

The value of P4CLIENT. May also be set in the environment. This is a required value.

--p4template <template>

The name of a Perforce client used to create a base snapshot before overlaying local changes. Defaults to the value of --p4client if not specified.

--p4changelist <change>

The changelist number (or default) whose changes are being tested. May be specified multiple times. If no changelists are specified, all client changelists will be used.

The following is a sample XML configuration file with all options specified:

   <?xml version="1.0" encoding="utf-8"?>
     <data>
       <server>
         <userName>myUser</userName>
         <password>myPass</password>
         <hostName>CloudBees FlowServer</hostName>
         <port>1234</port>
         <securePort>2345</securePort>
         <stompPort>3456</stompPort>
         <secure>0</secure>
         <timeout>3456</timeout>
             <driverLocation>/server/ec_preflight/alternateDrivers</driverLocation>
             <mainDriver>myMainClientDriver</mainDriver>
         </server>
         <procedure>
           <projectName>myProject</projectName>
           <procedureName>myProcedure</procedureName>
           <parameter>
               <name>branch</name>
               <value>main</value>
           </parameter>
           <parameter>
               <name>preflight</name>
               <value>true</value>
           </parameter>
           <priority>high</priority>
           <waitForJob>1</waitForJob>
           <jobTimeout>3600</jobTimeout>
         </procedure>
         <scm>
           <type>perforce</type>
           <port>perf:1234</port>
           <user>myUser</user>
           <password>myPass</password>
           <client>myUser-main-client</client>
           <template>myUser-main-template</template>
           <changelist>default</changelist>
           <changelist>67382</changelist>
           <autoCommit>1</autoCommit>
           <commitComment>Fixing bug 38582.</commitComment>
         </scm>
       </data>

In most cases, a developer does not need to override all default values and will probably want to specify passwords and some SCM-specific options on the command-line. The following is a more typical XML configuration:

   <?xml version="1.0" encoding="utf-8"?>
     <data>
       <server>
         <userName>myUser</userName>
         <hostName>commanderServer</hostName>
       </server>
       <procedure>
         <projectName>myProject</projectName>
         <procedureName>myProcedure</procedureName>
         <parameter>
            <name>branch</name>
            <value>main</value>
         </parameter>
         <parameter>
            <name>preflight</name>
            <value>true</value>
         </parameter>
         <jobTimeout>3600</jobTimeout>
       </procedure>
       <scm>
            <type>perforce</type>
            <port>perf:1234</port>
            <user>myUser</user>
            <client>myUser-main-client</client>
       </scm>
     </data>

If this file is stored in a .preflight file at the top-level of a source tree, the command to start the preflight would look something like this:

ecclientpreflight --p4changelist 56793 --autoCommit 1 --commitComment "Fixing bug 38582."

Default SCMs

CloudBees Flow includes preflight support for some of the most common Source Control (SCM) systems. For an SCM plugin you installed, look for a Help link for that plugin (on the Plugins page). Command-line options are included in this Help topic for the following SCMs:

  • AccuRev

  • Bazaar

  • ClearCase

  • CVS

  • Git

  • Mercurial

  • Perforce

  • Repo

  • StarTeam

  • Subversion

  • TFS

  • Vault

Perforce

The command-line options for the Perforce driver for ecclientpreflight are:

Perforce Options / Descriptions

--p4port <port>

The value of P4PORT may be set in the environment also.

--p4user <user>

The value of P4USER may be set in the environment also.

--p4passwd <password>

(Optional) The value of P4PASSWD may be set in the environment also.

--p4client <client>

The value of P4CLIENT may be set in the environment also.

--p4template <template>

(Optional) The name of a Perforce client used to create a base snapshot before overlaying local changes. Defaults to the value of ` --p4client` if not specified.

--p4changelist <change>

(Optional) The changelist number (or default) whose changes are being tested and can be specified multiple times. If no changelists are specified, all client changelists will be used.

--p4synctochange <change>

(Optional) The changelist number the Preflight Job should use when sync’ing the source tree. Values are: head —The most recent changelist anywhere in the P4 depot (default). have —The changelist for the most recent file that was synced to 'p4client' changelist. This is a p4changelist number.

Subversion

The command-line options for the Subversion driver for ecclientpreflight are:

Subversion Options / Descriptions

--svnpath <path>

The path to the locally accessible source directory in which changes were made.

--svnupdatetohead

Use this option to update the agent workspace created during preflight — updated to HEAD. By default, the agent workspace is updated to the revision found in the client workspace.

--svnignoreexternals

Causes the preflight process to ignore svn externals.

AccuRev

The command-line options for the AccuRev driver for ecclientpreflight are:

AccuRev Options / Descriptions

--accurevuser <user>

(Optional) The value of ACCUREVUSER may be set in the environment also. If not specified, the default is to the CloudBees Flow user.

--accurevpasswd <password>

The value of ACCUREVPASSWD may be set in the environment also. This is a required value.

--accurevpending <pending>

(Optional) Use this option to scan the client workspace for all pending elements using the "stat -fn -p" command. By default, the workspace is scanned for all kept elements using the "stat -fn -k" command.

--accurevpath <path>

The value of the ACCUREVPATH. Also may be set in the environment.

ClearCase

Preflight support for ClearCase is available only for snapshot views. The command-line options for the ClearCase driver for `ecclientpreflight ` are:

ClearCase Options / Descriptions

--ccpath <path>

The path to the locally accessible source directory where changes were made.

--ccUnixCSpecPath <unixCSpecPath>

The path to the appropriate config spec on a UNIX platform for the current view.

--ccWinCSpecPath <winCSpecPath>

The path to the appropriate config spec on a Windows platform for the current view.

--ccUnixRelativePath <unixRelativePath>

The relative path for the current view [on a UNIX platform] from the view root to the directory where changes were made.

--ccWinRelativePath <winRelativePath>

The relative path for the current view [on a Windows platform] from the view root to the directory where changes were made.

Bazaar

The command-line options for the Bazaar driver for ecclientpreflight are:

Bazaar Options / Descriptions

--workdir <path>

The developer’s source directory.

--branch <name>

Branch name for the preflight.

--method=<local|remote>

local— get tracked and untracked changes in the current workingdir remote— get changes between working tree and remote branch

Git

The command-line options for the Git driver for ecclientpreflight are:

Git Options / Descriptions

--gitdir=dir

The Git directory to process.

--method=<local_all | local_tracked | remote>

  • ` local_all` —get tracked and untracked changes between working tree and local repo

  • ` local_tracked` —get tracked changes between working tree and local repo

  • remote

CVS

The command-line options for the CVS driver for ecclientpreflight are:

CVS Options / Descriptions

--cvsroot <path>

The path to the repository.

--module <module>

The module name for the preflight.

--workdir <path>

The developer’s source directory.

Mercurial

The command-line options for the Mercurial driver for ecclientpreflight are:

Mercurial Options / Descriptions

--hgpath <path>

The path to the locally accessible source directory in which changes were made. Generally, this path is to the root of the workspace.

Repo

The command-line options for the Repo driver for ecclientpreflight are:

Repo Options / Descriptions

--repoworkdir <path>

The developer’s source directory.

--agentworkdir <path>

The path to the source directory, used by the agent.

StarTeam

The command-line options for the StarTeam driver for ecclientpreflight are:

StarTeam Options / Descriptions

--STProjectName <project>

The StarTeam project name.

--workingdir <path>

Working dir for the updated files.

TFS

The command-line options for the Team Foundation Server driver for `ecclientpreflight ` are:

TFS Options / Descriptions

--server <url>

The equivalent value to Collection in VS2008 and under. If you use this option, it is assumed you are using TF version VS2008 or under and it will use the option /server in the preflight. This field is required if you have VS2008.

--collection <url>

The URL that points to /collection. This value is used for VS2010. If you specify this value, the command uses the option /collection when it executes the TF query for preflight. This field is required if you have VS2010.

--localfolder

The path to the locally accessible source directory in which changes were made. Generally, this is the path to the root of the workspace.

--workspace <workspace>

The workspace containing the data for the preflight.

Vault

The command-line options for the Vault driver for ecclientpreflight are:

Vault Options / Descriptions

--workingdir <path>

Working dir for the updated files.

Troubleshooting

Client

The --debug option to `ecclientpreflight ` provides additional information about internal actions taken when starting the preflight build. By modifying the properties containing driver scripts, debug output can be added where needed.

If you want to capture debug information to a log file, use the ` --log` option. By default, logs are stored in the user’s home directory. To change this location, use the ` --logDir` option.

Agent

The step log for the custom preflight step provides information about the snapshot being created and the files being overlayed. Debug output can be added where needed by modifying the properties containing the driver scripts.