Make Compatibility

When GNU Make or NMAKE invokes a makefile target command, that process inherits the standard I/O streams of the Make process. It is then possible to invoke commands that expect input during a build, either from the terminal or passed into the Make standard input stream. For example, a makefile such as:

could be invoked like this:

More commonly, a command in a makefile might prompt the user for some input, particularly if the command encounters an error or warning condition:

Neither of these constructs is generally recommended because systems that require runtime user input are tedious to invoke and extremely difficult to automate.

Because Accelerator runs commands on cluster hosts where processes do not inherit the I/O streams and console of the parent eMake, makefiles (such as the examples above) with commands expecting interactive input are not supported.

In the majority of cases, tools that prompt for console input contain options to disable interactive prompting and proceed automatically. For example, invoking “ rm ” without the “ i ” enables this behavior. For those that do not, explicitly feeding expected input (either from a file or directly by shell redirection or piping) will suffice. For example:

Finally, tools such as Expect (see https://www.nist.gov/services-resources/software/expect ) that automate an interactive session can be used for commands that insist on reading from a console.

eMake simulates a serial GNU Make or NMAKE build. Though eMake runs many commands in parallel, the command output (including text written to standard streams and changes to the file system, such as creating or updating files) appears serially in the original order without overlapping. This feature is called transactional output (or “serial order execution”) and is unique to Accelerator among parallel build systems. This feature ensures standard output streams and underlying file systems always reflect a consistent build execution state, regardless of how many jobs eMake is actually running concurrently on the cluster .

Transactional output is achieved by buffering the results of every command until the output from all preceding commands is written. Buffering means that while the output contents on the standard streams matches GNU Make or NMAKE exactly, the timing of its appearance might be a little unexpected. For example:

For example, consider a rule that uses rsync to deploy output:

With GNU Make, users first see the Copying output echo, then the state information from rsync as it builds the file list, copies files, and finally, they see the done echo as the job completes.

With eMake, all output from this job step appears instantaneously in one burst when the job completes. By the time any output from echo or rsync is visible, the entire job has completed.

Recursive Makes (also called submakes because they are invoked by a parent or top-level Make instance) are often used to partition a build into smaller modules. While submakes simplify the build system by allowing individual components to be built as autonomous units, they can introduce new problems because they fragment the dependency tree. Because the top-level Make does not coordinate with the submake—it is just another command—it is unable to track targets across Make instance boundaries. For a discussion of submake problems, see “Recursive Make Considered Harmful” by Peter Miller (https://www.scribd.com/document/15677619/Recursive-Make-Considered-Harmful).

Submakes are particularly problematic for parallel builds because a build composed of separate Make instances is unable to control target serialization and concurrency between them. For example, consider a build divided into two phases: a libs submake that creates shared libraries followed by apps that builds executables that link against those libraries. A typical top-level makefile that controls this type of build might look like this:

This type of makefile works fine for a serialized make, but running in parallel it can quickly become trouble:

which can make it difficult for a Make system to identify submake invocations, let alone attempt to ensure their correct, concurrent execution. These problems are exacerbated with distributed (cluster) parallel builds because each make invocation is running on a remote Agent.

Correct, highly concurrent parallel builds require a single, global dependency tree. Short of re-architecting a build with submakes into a single Make instance, this is very difficult to achieve with existing Make tools.

An ideal solution to parallelizing submakes has the following properties:

eMake differs from other Make variants in the way it searches for files needed by pattern rules (also called suffix or implicit rules) in a build.

Because of the difference in the way eMake and NMAKE match pattern rules, NMAKE and eMake can produce different makefile output with hidden targets . A hidden target (also known as a “hidden dependency”) is a file that is:

Consider the following makefile example:

Notice that foo.c is created as a side effect of updating the bar.lib target. Until bar.lib is updated, no rule is available to update foo.obj because nothing matches the .c.obj suffix rule.

NMAKE accepts this construct because it checks for foo.c existence before it attempts to update foo.obj. NMAKE produces the following result for this makefile:

eMake, however, performs the search for files that match the suffix rule once so it can schedule all jobs immediately and maximize concurrency. eMake will not notice the existence of foo.c by the time it attempts to update foo.obj, even if foo.c was created. eMake fails with:

The fix is simply to identify foo.c as a product for updating the bar.lib target, so it is no longer a hidden target. For the example above, adding a line such as foo.c: bar.lib is sufficient for eMake to understand that .c.obj suffix rule matches the foo.obj target if bar.lib is built first. Adding this line is more accurate and has no effect on NMAKE.

GNU Make is similarly incompatible with eMake, but the incompatibility is sometimes masked by the GNU Make directory cache. GNU Make attempts to cache the directory contents on first access to improve performance. Unfortunately, because the time of first directory access can vary widely depending on which targets reference the directory and when they execute, GNU Make can appear to fail or succeed randomly in the presence of hidden targets.

For example, in this makefile, the file $(DIR)/foo.yy is a hidden target created as a side-effect of updating aa and needed by the pattern rule for foo.xx :

Depending on the value of DIR, this build might or might not work with GNU Make:

eMake does not attempt to emulate this behavior. Instead, it consistently refuses to schedule foo.xx because it depends on a hidden target (just as it did in the NMAKE emulation mode in the earlier example). In this case, adding a single line declaring the target: $(DIR)/foo.yy: aa is sufficient to ensure it always matches the %.xx pattern rule.

There are many other reasons why hidden targets are problematic for all Make-based systems and why eliminating them is good practice in general. For more information, see:

A number of differences exist between GNU Make and eMake regarding the use of $(wildcard) and prerequisite wildcard sort order functions. When using the $(wildcard) function or using a wildcard in the rule prerequisite list, the resultant wildcard sort order might be different for GNU Make and eMake.

Different GNU Make versions are not consistent and exhibit permuted file lists. Even a GNU Make version using different system libraries versions will exhibit inconsistencies in the wildcard sort order.

No difference exists in the file list returned, other than the order. If the sort order is important, you might wrap $(wildcard) with $(sort).

For example:

Do not rely on the order of rule prerequisites generated with a wildcard. For example, using target: *.foo.

Relying on the order of *.foo can be dangerous for both GNU Make and eMake. Neither GNU Make nor eMake guarantees the order in which those prerequisites are executed.

All Make variants process makefiles by looking for rules to build targets in the dependency tree. If no target rule is present, Make looks for a file on disk with the same name as the target. If this existence check fails, Make notes it has no rule to build the target.

eMake also exhibits this behavior, but for performance reasons it delays the existence check for a target without a makefile rule until just before that target is needed. The effect of this optimization is that eMake might run more commands to update targets (than GNU Make or NMAKE) before it discovers it has no rule to make a target.

For example, consider the following makefile:

GNU Make begins by looking for a rule for nonexistent and, when it does not find the rule, it does a file existence check. When that fails, GNU Make terminates immediately with:

Similarly, NMAKE fails with:

eMake delays the existence check for nonexistent until it is ready to run the all target. First, eMake finishes running commands to update the aa and bb prerequisites. eMake fails in the same way, but executes more targets first:

Of course, when the existence check succeeds (as it does in any successful build), there is no behavioral difference between eMake and GNU Make or Microsoft NMAKE.

GNU Make has an advanced feature called Makefile Remaking, which is documented in the GNU Manual, “How Makefiles are Remade,” and available at: https://www.gnu.org/software/make/manual/make.html#Remaking-Makefiles

To quote from the GNU Make description:

“Sometimes makefiles can be remade from other files, such as RCS or SCCS files. If a makefile can be remade from other files, you probably want make to get an up-to-date version of the makefile to read in.

“To this end, after reading in all makefiles, make will consider each as a goal target and attempt to update it. If a makefile has a rule which says how to update it (found either in that very makefile or in another one) or if an implicit rule applies to it (see section Using Implicit Rules), it will be updated if necessary. After all makefiles have been checked, if any have actually been changed, make starts with a clean slate and reads all the makefiles over again. (It will also attempt to update each of them over again, but normally this will not change them again, since they are already up to date.)”

This feature can be very useful for writing makefiles that automatically generate and read dependency information with each build. However, this feature can cause GNU Make to loop infinitely if the rule to generate a makefile is always out-of-date:

In practice, a well-written makefile will not have out-of-date rules that cause it to regenerate. The same problem, however, can occur when Make detects a clock skew—most commonly due to clock drift between the system running Make and the file server hosting the current directory. In this case, Make continues to loop until the rule to rebuild the makefile is no longer out-of-date.

In the example below, DIR1 and DIR2 are both part of the source tree:

If two directories are served by different file servers and the clock on the system hosting DIR2 is slightly faster than DIR1, then even though foo.dd is updated after bar.dd, it might appear to be older. On remaking, GNU Make will again see foo.dd as out-of-date and restart, continuing until the drift is unnoticeable.

This problem is particularly troublesome on Solaris, where GNU Make timestamp checking has nanosecond resolution:

eMake fully supports makefile remaking and can be configured to behave exactly as GNU Make. However, by default, to ensure builds do not loop unnecessarily while remaking, eMake limits the number of times it restarts a make instance to 10. If your build is looping unnecessarily, you might want to lower this value or disable remaking entirely by setting:

NMAKE contains a feature to create inline files with temporary file names. For example, the following makefile creates a temporary inline file containing the word “pass” and then uses “type” to output it.

With Microsoft NMAKE, the file is created in the directory where %TMP% points. eMake does not respect the %TMP% setting and creates the inline file in the rule working directory that needs the file.

eMake uses the following process: