This chapter provide information about how CloudBees Build Acceleration integrates with ClearCase.
If your build environment relies on the ClearCase source control system, there are some special considerations for running eMake. ClearCase views can be either “snapshot” or “dynamic.”
-
ClearCase snapshot views behave like a normal file system, so no special support is required.
-
ClearCase dynamic views have non-standard file system behavior that requires explicit handling by eMake.
ecclearcase executable
Set the EMAKE_CLEARCASE_SERVER environment variable to the path of the ecclearcase executable. This ensures eMake can locate the correct version of ecclearcase if, for example, you are using 64-bit eMake.
LD_LIBRARY_PATH
Using CloudBees Build Acceleration in a ClearCase environment requires your LD_LIBRARY_PATH (on UNIX) or PATH (on Windows) to contain a directory that includes libraries required to run cleartool. Library file names for Windows or UNIX begin with libatria (Windows - libatria.dll, UNIX - libatria.so).
If you plan to use ClearCase with eMake, you must add the ClearCase shared libraries to the LD_LIBRARY_PATH on your system.
For sh:
LD_LIBRARY_PATH=/usr/atria/linux_x86/shlib:$LD_LIBRARY_PATHexport LD_LIBRARY_PATH
For csh:
setenv LD_LIBRARY_PATH /usr/atria/linux_x86/shlib:${LD_LIBRARY_PATH}
( /usr/atria/linux_x86/shlib is an example and might differ on your system depending on what OS you use and where ClearCase is installed.)
To ensure CloudBees Build Acceleration knows where ClearCase is installed, edit /etc/ld.so.conf to include the ClearCase installation location. As a second option, you can include the ClearCase installation location in LD_LIBRARY_PATH.
ClearCase views on agents
When CloudBees Build Acceleration replicates a ClearCase view on an agent, it appears as a generic file system—ClearCase commands that run as part of a build will not work on the host, even if ClearCase is installed on that machine. The Electric File System masks ClearCase’s VOB mounts. If your build runs ClearCase commands, these commands must be runlocal steps. For additional information, see Running a Local Job on the Make Machine.
| Because of the potential adverse interaction between two different file systems (ClearCase and CloudBees Build Acceleration), CloudBees recommends that you do not install ClearCase on agent machines. If you must run ClearCase on an agent machine, ensure that whichever one you need to start and stop frequently is configured to start “second” at system startup time. |
--emake-clearcase
The eMake command-line option --emake-clearcase controls which ClearCase features are supported for a build. By default, ClearCase integration for rofs, symlink, and vobs is disabled. To turn on support for these specific ClearCase features, if your build relies on these options, use --emake-clearcase=LIST, where LIST is a comma-separated list of one or more of the following values:
-
rofs: detect read-only file systems
eMake queries ClearCase for each file it accesses to determine whether the file should be considered 'read-only'.
-
symlink: detect symbolic links (Windows only)
eMake queries ClearCase for each file it accesses to determine whether the file is a ClearCase symbolic link.
-
vobs: configure separate temporary directories for each vob
Normally, eMake uses the --emake-tmpdir setting to determine where to place temporary directories for each device. With the vobs option enabled, eMake automatically configures one directory per VOB. On Windows, eMake also communicates with ClearCase to determine which VOB a file belongs to so it can select the correct temporary directory.
| If `--emake-clearcase ` is not specified on the command line and the environment variable `EMAKE_CLEARCASE ` is present, eMake takes the options from the environment. |
eMake’s "fake" interface for ClearCase
In addition to a direct interface to ClearCase, eMake also provides a “fake” interface that allows the end user to pass information manually to eMake about the ClearCase environment. Normally, you invoke ClearCase functionality by specifying --emake-clearcase=LIST to eMake, at which point eMake attempts to load ecclearcase6.so and ecclearcase7.so (.dll on Windows). Whichever library successfully initializes in the ClearCase environment is used to talk to ClearCase through a provided API that is no longer maintained or supported. You can specify the precise library to load by setting the environment variable EMAKE_CLEARCASE_LIBRARY to the path to the desired library.
Under some conditions, the ClearCase API does not function properly. For this circumstance, eMake provides ecclearcase_fake.so (.dll on Windows). If you point EMAKE_CLEARCASE_LIBRARY to the fake interface, eMake loads that instead. The fake interface then loads the file specified in the environment by ECCLEARCASE_FAKE_INI, defaulting to ecclearcase_fake.ini. The ini file has two sections: [vobs] and [attrs].
The [vobs] section maps a VOB path to a comma-separated set of attributes. Currently, public should be present for a public VOB and ro for read-only.
The [attrs] section maps a file name to symlink * type , where symlink might be empty if the file is not a symbolic link and type can be null, version, directory_version, symbolic_link, view_private, view_derived, derived_object, checked_out_file, checked_out_dir. If symlink is not empty, symbolic_link ` is assumed. If type is `version or directory_version and --emake-clearcase=rofs is active, the EFS returns EROFS (or STATUS_ACCESS_DENIED on Windows) when an attempt is made to write the file.
If CLEARCASE_ROOT is set in the environment (as by cleartool setview ), all [attrs] entries are tracked under their exact path as well as one with the CLEARCASE_ROOT prepended. If CLEARCASE_ROOT is set to /view/testview, setting /vobs/test/symlink2 in [attrs] is the same as setting both /vobs/test/symlink2 and /view/testview/vobs/test/symlink2.
Sample ini files for UNIX:
[vobs] /vobs/test=public /vobs/readonly=public,ro
[attrs] /vobs/test/symlink2=symlink /vobs/test/symlink/alpha=*directory_version /vobs/test/symlink/beta=alpha
and Windows:
[vobs] \test=public \readonly=public,ro
[attrs] S:/test/symlink2=symlink S:/test/symlink/alpha=*directory_version S:/test/symlink/beta=alpha
Where ClearCase dynamic views affect eMake behavior
Read-only mounts
ClearCase can mount files in a read-only mode, which means they appear to be writable, but any attempts to modify these files fail with a “read-only file system” (UNIX) or “access denied” (Windows) error message. Because eMake cannot tell whether a file is modifiable using normal file system interfaces, it does not know to disallow modifications performed by commands running on the agents. This activity leads to failures when eMake attempts to commit changes (incorrectly) allowed on the agent. A simple test case:
unix% cleartool lsMakefileclock@@/main/2 Rule: /main/LATEST
unix% cat Makefileall: mv clock clock.old
The file “clock” is checked in to ClearCase. Makefile attempts to rename it. If you just run “make”, it fails immediately, but can be instructed to ignore the error:
unix% make -imv clock clock.oldmv: cannot move `clock` to `clock.old`: Read-only file systemmake: [all] Error 1 (ignored)
Note that this file system is not mounted read-only, so a Makefile can be created. Because “clock” is checked-in, it cannot be renamed without checking it out first, and ROFS is the error ClearCase gives.
Now try this with eMake:
unix% emake --emake-root=/vobs -i Starting build: 114626 mv clock clock.old ERROR EC1124: Unable to rename file/vobs/test/drivel/clock to /vobs/test/drivel/clock.old: Read-onlyfile system (error code 0x1e): Read-only file systemInterrupted build: 114626 Duration: 0:00 (m:s) Cluster availability: 100%
Without activating ClearCase support, eMake does not know “clock” cannot be moved, so the operation succeeds on the agent, then fails when eMake attempts to commit it to disk. Specifying the “ -i ” flag to ignore errors will not work here.
unix% /home/user/Projects/4.2/i686_Linux/ecloud/emake/emake --emake-clearcase=rofs --emake-root=/vobs -i Starting build: 114630 mv clock clock.old mv: cannot move `clock` to `clock.old`: Read-only file system make: [all] Error 1 (ignored) Finished build: 114630 Duration: 0:00 (m:s) Cluster availability: 100%
When eMake knows to replicate ClearCase’s behavior, the error occurs on the host and can be handled normally.
Multiple VOBs
eMake writes uncommitted files into temporary directories, and moves them into their correct location after resolving any conflicts. eMake automatically places a temporary directory in the current working directory where it is invoked, and also creates a temporary directory in each location specified by the --emake-tmpdir option or the EMAKE_TMPDIR environment variable. When possible, eMake writes uncommitted files to the same physical device where the file will be saved when it is committed, which makes the commit operation a lightweight “rename” instead of a heavyweight “copy” operation.
Under ClearCase, each VOB functions as a separate physical disk, so to achieve optimal performance, a temporary directory must be specified for each VOB where the build writes files. --emake-clearcase=vobs sets up this directory for you automatically.
-
On UNIX, each VOB has a distinct physical device ID, and this option is nothing more than a “shorthand” for specifying
EMAKE_TMPDIR=/vobs/foo:/vobs/bar:….in the environment. -
On Windows, you must interface with ClearCase directly to make this distinction, so using
--emake-clearcase=vobsis important to get the most speed for a build that writes to multiple VOBs.
Symbolic links on Windows platforms
On Windows, ClearCase conceals the nature of its symbolic links from other programs, so what is actually a single file appears to be two files to other programs. This situation hinders eMake’s versioning mechanism as it tracks two separate chains of revisions for one underlying entity. A job’s view of the file can get out of sync and cause build failures.
--emake-clearcase=symlink interfaces directly with ClearCase to determine whether a particular ClearCase file is a symbolic link and represents it on the agent as a reparse point, which is the native Windows equivalent of a symbolic link. All file operations are redirected to the target of the symbolic link to avoid synchronization problems. T
This issue does not occur on UNIX platforms, because ClearCase uses native file system support for symbolic links.
Following is a simple test case.
Beginning with a directory, “alpha”, and a symlink to that directory, “beta”:
windows% cleartool ls -d alpha betaalpha@@/main/1 Rule: \main\LATESTbeta --> alpha
And a makefile:
all: @echo "Furshlugginer" > alpha/foo @echo "Potrzebie" > beta/foo @cat alpha/foo
windows% emake --emake-root=. -f symlink.mk Starting build: 50070 FurshlugginerFinished build: 50070 Duration: 0:02 (m:s) Cluster availability: 100%
windows% emake --emake-root=. -f symlink.mk --emake-clearcase=symlink Starting build: 50071 PotrzebieFinished build: 50071 Duration: 0:01 (m:s) Cluster availability: 100%
Explanation: ClearCase cannot tell the Windows file system that the symlink is a symlink, so alpha/foo and beta/foo appear to be distinct files. (On UNIX, this is not an issue, because symlinks are a standard operating system feature, which means that ClearCase can show them as such.) If a build does not contain ClearCase symbolic links, there is no reason to turn on the integration; if it does, eMake might assume that two different files exist when there is actually just one underlying file, in which case you must turn on the “symlink” part of the CloudBees Build Acceleration ClearCase integration.
Performance considerations
Running builds from ClearCase dynamic views can impose a considerable performance cost depending on the ClearCase configuration and your build. The best performance is achieved by using ClearCase snapshot views. If using snapshots is not possible, there are a few things to consider when setting up an eMake build.
Enabling the “symlink” or “rofs” options incurs a performance cost because of the need to communicate with the ClearCase server when accessing a file for the first time. Many builds do not need these features, even if they are running inside a ClearCase dynamic view, so consider leaving them turned off unless you encounter unexpected build failures.
Enabling the “vobs” option should have minimal performance cost, and might significantly speed up your build if build output is written back to your dynamic view.
Because of improved caching, eMake might provide a significant performance boost beyond that provided by running build steps in parallel. eMake caches much of the file system state, reducing the total number of requests to the ClearCase server during the build. Depending on how heavily loaded your ClearCase server is, this can significantly improve build performance. If you notice build speedups higher than you would expect given the number of agents in your cluster, improved caching might be the reason.
Using the “fake” interface for ClearCase (see eMake’s "fake" interface for ClearCase, which lets you specify the details of VOBs and files in a static file, is much faster than communicating with ClearCase. This might suffice for many users.