CloudBees Build Acceleration eDepend

In its default configuration, CloudBees Build Acceleration is designed to be a drop-in replacement for your existing Make tool—GNU Make or Microsoft NMAKE. CloudBees Build Acceleration behaves exactly like your existing Make: it will rebuild (or declare up-to-date) the same targets following the same rules Make uses. The file dependency tracking technology in the Electric File System (EFS) and the eMake history feature is used to ensure the system reproduces exactly the same results in a parallel, distributed build as it would serially.

Because the system captures and records such detailed information about the relationships between build steps, it is uniquely capable of accomplishing much more than simply ensuring parallel builds are serially correct. In particular, by enabling CloudBees Build Acceleration eDepend, you can wholly replace tools and techniques like makedepend or gcc -M —commonly used to generate makefiles too difficult to maintain by hand (for example, C-file header dependencies).

CloudBees Build Acceleration eDepend is easier to configure, faster, more accurate, and applicable to a much wider range of dependencies than existing dependency generation solutions. If your current build does not employ dependency generation, you can enable eDepend and benefit from more accurate incremental builds without the overhead of configuring and integrating an external dependency generation tool.

The following sections describe the dependency generation challenge in more detail and how eDepend can improve your build speed and accuracy.

Dependency generation

Consider a build tree that looks like this:

src/▼

   Makefile▼

←-- top-level makefile: recurses into mylib and then into main to build the program

   common/▼

      header1.h▼

      header2.h▼

   mylib/▼

      Makefile▼

←-- has rules to build mylib.o and create library mylib.a

      mylib.h▼

      mylib.c▼

←-- includes common/header1.h and mylib.h

   main/▼

      Makefile▼

←-- has rules to build main.o and then to link main using main.o and mylib.a

      main.c▼

←-- includes common/header1.h, common/header2.h, and lib/mylib.h

The problem

Even in this simple example, the need for dependency generation is apparent: if you make a change to a header file, how do you ensure dependent objects are recompiled when you rebuild?

Makefiles could explicitly declare all header dependencies, but that quickly becomes too cumbersome: each change to an individual source file might or might not require an adjustment in the makefile. Worse, conditionally compiled code can create so many permutations that the problem becomes intractable.

Another possibility is to declare all headers as dependencies unilaterally, but then the build system becomes very inefficient: after a successful build, a modification to header2.h should trigger a rebuild only of the main module, not mylib.a as well.

Clearly, to get accurate, efficient builds, the system must have calculated dependencies automatically before it builds.

There are several ways to generate dependencies and update makefiles to reflect these dependencies (for example, makedepend or gcc -M ), but they all have the drawbacks mentioned previously.

eDepend benefits

CloudBees Build Acceleration eDepend is an eMake feature that directly addresses all problems with existing dependency generation solutions. Specifically:

It is part of eMake and requires no external tool to configure, no extra processing time, and it is faster than other solutions.
It is easily enabled by setting a command-line parameter to eMake. No tools or changes to makefiles are required.
Like CloudBees Build Acceleration itself, it is completely tool and language independent. eDepend automatically records any and all kinds of dependencies, including implicit relationships such as executables on libraries during a link step.
It is completely tool and language independent. eDepend automatically records any and all kinds of dependencies, including implicit relationships such as executables on libraries during a link step.
eDepend dependencies are recorded in eMake history files—transparently recorded and used without manifesting as makefile rules.
eDepend is accurate because it uses file information discovered by the Electric File System at the kernel level as a job executes its commands.

How does eDepend work?

Internally, eDepend is a simple application of sophisticated file usage information returned by the Electric File System.

As a job runs, the Electric File System records all file names it accesses inside EMAKE_ROOT.

This function has two very important implications:

eDepend can track dependencies within EMAKE_ROOT only.
eDepend can track dependencies for a job only after it has run—this is why you must start with a complete build rather than an incremental build.
1. After a job completes, eMake saves the following eDepend information to the eMake history file:
the working directory for the Make instance
the target name
any files actually read (not just checked for existence) and/or explicitly listed as prerequisites in the makefile

eDepend information is stored in the history file along with serialization history information. Commands operating on the history file (for example, those specifying file location or that erase it) apply to eDepend information as well.

In a subsequent build, whenever eMake schedules a target in a directory for which it has eDepend information, it evaluates file dependencies recorded in the earlier run as it checks to see if the target is up-to-date.

In the example above, the rule to update mylib.o might look like this:

mylib.o: mylib.c
    $(CC) ...▼

mylib.c includes common/header1.h, which is not explicitly listed as a prerequisite of mylib.o, so eDepend records this implicit dependency in the history file.

Directory	Object	Dependency
src/mylib	mylib.o	common/header1.h

Enabling eDepend

Start from a clean (unbuilt) source tree.
If your build system already has a dependency generation mechanism, turn it off if possible. If you cannot turn it off, you will still get eDepend’s additional accuracy, but you will not be able to improve the performance or shortcomings of your existing system.
Build your whole source tree with eDepend enabled.

Use the --emake-autodepend=1 command-line switch:
```
% emake --emake-autodepend=1 ...▼
```
Alternatively, insert --emake-autodepend=1 into the EMAKEFLAGS environment variable.
```
% setenv EMAKEFLAGS --emake-autodepend=1
% emake ...▼
```
Make a change to a file consumed by the build, but not listed explicitly in the makefiles.

For example, touch a header file: % touch someheader.h
Rebuild, again making sure eDepend is enabled.
```
% emake --emake-autodepend=1 ...▼
```
Notice that without invoking a dependency generator, eMake detected the changed header and rebuilt accordingly.

Important notes

The eDepend list is consulted only if all other prerequisites in the makefile indicate the target is up-to-date.

Explained another way: If a target is out-of-date because it does not exist or because it is older than one of the prerequisites listed in the makefile, eDepend costs nothing and has no effect.

If the target is newer than all its listed prerequisites, then eDepend is the “11th hour” check to ensure it really is up-to-date, and that there is not a newer implicit dependency. This is the only place eDepend interacts with your build: it forces a target that incorrectly appears to be up-to-date to be rebuilt.

eDepend information, unlike traditional Makedepend rules, does not in any way imply anything about needing to build or update the implicit prerequisite.

In the example above, if header1.h is renamed or moved, eMake just ignores the stale eDepend information. When eMake next updates the mylib.o target, it will prune stale dependencies from the eDepend list. This change to the history file occurs regardless of the setting of the --emake-history-force parameter to eMake.

Unlike Make, eMake does not complain if it does not have a rule to make header1.h because eDepend dependencies are not used to schedule targets.

eMake considers a rule to be out of date when an implicit dependency is removed. This causes eMake to rebuild the target, matching ClearMake.
eDepend’s information scope is bound by a directory name and the target name. This means you can build cleanly from the top of a tree, then run accurate incremental builds from individual subdirectories and eDepend information will be used and updated correctly.

However, it does imply if you have a build that

performs multiple passes or variants over the same directories
with exactly the same target names, but
runs significantly different commands

For example, a build that produces objects with the same name for a different architecture or configuration, eDepend information might be over-used unnecessarily. In this case, eMake might rebuild more than is necessary, but with no incorrect build results. In this situation, you can achieve fast, correct builds by using separate history files, or ideally, by changing to unique target names across build variants.

Using #pragma noautodep

Some build steps contain many implicit dependencies that might not make sense to check for up-to-dateness. Examples include symbol files consumed by a link step or archive packager input files (for example, tar). In both cases, any makefile explicit prerequisites are sufficient to determine if the target should be updated: eDepend information would just add overhead or cause unnecessary rebuilds.

You can selectively disable eDepend information for certain files from any step by supplying eMake with the makefile directive:

#pragma noautodep *.pdb
%.o: %.c
    $(CL) ...▼

The directive #pragma noautodep is applied to the next rule or pattern rule in the makefile. This directive specifies a class of files for eDepend to ignore. Note the following information about #pragma noautodep:

Wildcards allowed for #pragma noautodep:
- matches 0 or more characters
  
  ? matches 1 character
  
  [ ] matches a range of characters
The noautodep filters are matched against absolute path names. To limit a filter to files in the current directory for the job, use ./:
```
#pragma noautodep ./foo.h▼
```
To specify ignore foo.h in any directory, use:
```
#pragma noautodep */foo.h▼
```
If the supplied pattern has no wildcards and does not specify a path, it will never match.

eMake ignores the directive and prints a warning as it parses the makefile:
```
Makefile:2: ignoring autodep filter `foo`, does not match absolute path(s).▼
```

#pragma noautodep affects what is put into the eDepend database, not what is pulled out of it. In other words, if eDepend recorded a dependency on file bar for target foo, and you subsequently add #pragma noautodep ./bar, then the next time eDepend is updated for foo, bar will be excluded from the eDepend database.