When CloudBees Build Acceleration runs a build for the first time, it takes aggressive action to run all jobs as fast as possible in parallel. Jobs that run in the wrong order because of missing makefile dependencies are automatically re-run to ensure correct output. (These failed steps are also called conflicts . See Conflicts and Conflict Detection for information about conflicts.)
To avoid the cost of re-running jobs on subsequent builds, eMake saves the missing dependency information in a history data file . The history data file evolves with each new run and enables CloudBees Build Acceleration.
You can choose the location of the history file and how it is updated.
Setting the history file location
By default, eMake creates the history file in the directory you use to invoke the build and names it emake.data by default. The file location can be explicitly specified using the command-line option:
--emake-historyfile=<pathname>
The history file is used for two operations during an eMake cluster build:
-
Input: eMake reads the history file as it starts a build to improve build performance.
-
Output: eMake writes to the history file as it completes a build to improve performance of subsequent builds.
History file input rules
If the history file (emake.data or whatever was specified with --emake-historyfile) exists, it is always read and used to improve performance.
History file output rules
Data written to the history file after the build depends on the --emake-history option setting. Three options are available:
-
Merge: By default, eMake merges new dependencies into the existing history file. In this way, the history file evolves automatically as your makefiles change by learning dependencies that accelerate your builds.
-
Create: If
--emake-historyis set tocreate, the old history file contents are overwritten by new dependencies discovered in the run that just completed. Use this setting to start a fresh history file to eliminate stale information from the file. -
Read: If
--emake-historyis set toread, no data is written to the history file at build completion, and any new dependencies discovered are discarded. Use this setting when developers share a single, static copy of the history file.
By default, the history file is updated even if the build fails, regardless of the value of --emake-history. You can override this behavior by setting --emake-history-force=0.
The history file directly impacts the number of conflicts the build can encounter. Ideally, a CloudBees Build Acceleration cluster build with good history should have almost no conflicts. If conflicts are increasing, check for a current history file.
Guaranteeing correct history
Use --emake-readdir-conflicts=1 to guarantee correct history. Some parallel builds do not succeed without a good history file. In particular, builds that use wildcard or globbing operators to produce build-generated lists of files and operate on those lists might fail. For example, a makefile might use ld .o as shorthand to avoid enumerating all the .o files in a directory. Running the build with --emake-readdir-conflicts=1 guarantees that the build succeeds and that a history file is created for use by subsequent parallel builds.
Do not enable --emake-readdir-conflicts=1 all the time. Instead, enable it for one run if you suspect a globbing problem, and then disable it, but use the history file generated by the previous run.
You can alternatively use the #pragma readdirconflicts pragma to enable directory-read conflicts on a per-job basis. You can apply it to targets or rules in your makefiles. This pragma has less overhead than --emake-readdir-conflicts=1 (which enables directory-read conflicts for an entire build). You can use this pragma in pragma addendum files as well as in standard makefiles.
Ensuring that relative EMAKE_ROOT locations match
Relative EMAKE_ROOT locations must match. The history file records target file names relative to the EMAKE_ROOT specified during that run. For a subsequent build to use the history file correctly, target file names must have the same path name relative to the eMake root.
For example, if your eMake root is /home/alice/builds and your build references a path name in that root: /home/alice/builds/lib/foo.o, then the history file records it as lib/foo.o. If a subsequent build sets the eMake root to /home/bob/builds, the history file will match correctly.
If, however, the eMake root is /home/bob, then the file that exists on the disk as /home/bob/builds/lib/foo.o is assigned the root-relative name of builds/lib/foo.o, which does not match the name lib/foo.o in the history file generated above. Because the history file does not match, performance might suffer.
EMAKE_ROOT must match the same location relative to sources as the EMAKE_ROOT used to create the history file.
|
Running builds with multiple roots
For builds with multiple roots, the roots must have the same alphabetical sorting order in each build so that the history matches.
Using the remaphist utility to relocate a history file
The remaphist utility makes it easier for users to share history files. It is located at:
-
(Linux)
<install_dir>/i686_Linux/unsupported/remaphist, where<install_dir>is/opt/ecloudby default -
(Windows)
<install_dir>\i686_win32\unsupported\remaphist, where<install_dir>isC:\ECloudby default
Modes of operation
The remaphist utility has two modes of operation:
-
Maps a standard eMake history file to a “relocatable” (and therefore usable in other build environments) history file.
History files store paths in the form
<root_ID><root_relative_path>, so if your root is/home/stevecand you have a path such as/home/stevec/proj1, the history file records it as0 proj1. A relocatable history file flattens those references and then replaces the root prefixes with variables that are easier to “swizzle” later. For example, this step takes0 footo/home/stevec/proj1to$(ROOT0)/proj1. -
Converts a relocatable history file back into a standard history file.
This step expands the variables according to the new user’s specification and then converts them into new root-relative paths. For example, this step takes
$(ROOT0)/proj1to/home/tmurphy/footo0 foo.
Syntax
To remap a standard file to make it relocatable:
remaphist -i <input_file> -o <mapped_file> [-r <emake_root(s)>] <absolute_path1>=<VARIABLE1> [<absolute_path2>=<VARIABLE2> … ]
To unmap a relocatable file back to a standard file:
remaphist -u -i <mapped_file> -o <output_file> -r <emake_root(s)><VARIABLE1>=<absolute_path1> [<VARIABLE2>=<absolute_path2> … ]
Note that specifying the variables/paths to remap are specified as <absolute_path>=<VARIABLE> when remapping a standard history file, but are specified as <VARIABLE=<absolute_path> when unmapping a relocatable history file back to a standard one.
| Option | Description |
|---|---|
-u |
Performs an unmapping. The default is to perform a remapping. |
-i <input_file> |
Path to the unmapped (standard) input file. |
-o <mapped_file> |
Path to the remapped (relocatable) file to create. |
-i <mapped_file> |
Path to the remapped (relocatable) input file. |
-o <output_file> |
Path to the unmapped (standard) file to create. |
-r <emake_root(s)> |
Specifies the eMake root(s). |
-s <0|1> |
(Optional) Specifies whether to sort the root directories. The default is |
<absolute_path>=<VARIABLE> […] |
eMake roots to remap (not |
<VARIABLE>=<absolute_path> […] |
eMake roots to unmap ( |
-help or -? |
(Optional) Prints the help message. |
Examples
The following example shows how to remap a standard history file to make it relocatable:
remaphist -i emake.data -o remapped.data /home/kathy/abs/proj=PROJ /home/kathy/modules=MODULES
The following example shows how to unmap a remapped history file back to a standard file:
remaphist -u -i remapped.data -o emake.data2 -r /workspace/tools PROJ=/opt/kathy/abs/proj MODULES=/opt/kathy/modules/