Make Compatibility

19 minute read

ElectricAccelerator is designed to be completely compatible with existing Make variants it emulates. There are, however, some differences in behavior that might require changes to makefiles or scripts included in the build. Almost all GNU Make and NMAKE options are valid for use with ElectricAccelerator. However, ElectricAccelerator does not support some GNU Make and NMAKE options. The following topics documents those differences and what actions to take to ensure your build is compatible with ElectricAccelerator.

Unsupported GNU Make Options and Features

Unsupported GNU Make Options

Option eMake response if specified

-d (debug)

Error message

-j (run parallel)


-l (load limit)


-o (old file)

Error message

-p (print database)

Error message

-q (question)

Error message

-t (touch)

Error message

GNU Make 3.81 Support

eMake does not support the following GNU Make 3.81 features:

  • Though $(eval) is allowed in rule bodies, any variables created by the $(eval) exist only in the scope of the rule being processed, not at the global scope as in GNU Make. For example, the following is not supported:

    $(eval foo: bar)
  • Using ` $*` for an archive member target

GNU Make 3.82 Support

GNU Make 3.82 emulation supports the following GNU Make 3.82 features:

  • Multi-line variable definition types

  • Behavior in which pattern rule searches prefer the most-specific matching pattern, rather than the first matching pattern. For example, for a target {{foo.o}} and patterns {{%.o}} and {{f%.o}}, in that order, gmake 3.81 uses the {{%.o}} pattern, because it is the first match, but gmake 3.82 uses the {{f%.o}} pattern, because it is a more specific match

Other GNU Make 3.82 features are not supported.

GNU Make 4.0 and 4.1 Support

GNU Make 4.0 and 4.1 emulation supports the GNU Make 4.0 shell assignment feature, where VAR!=<command> is equivalent to VAR:=$(<shell command>). Other GNU Make 4.0 features are not supported.

GNU Make 4.2 Support

GNU Make 4.2 emulation supports the GNU Make 4.2 $(.SHELLSTATUS) variable. This variable is set to the exit status of the last != or $(shell …​) function invoked in this instance of Make. The value is 0 if successful or > 0 if not successful. The value is unset if the != or $(shell …​) function was not invoked. Other GNU Make 4.2 features are not supported.

Unsupported NMAKE Options

  • /C (Suppresses default output, including nonfatal NMAKE errors or warnings, timestamps, and NMAKE copyright message. Suppresses warnings issued by /K )

  • /T (Updates timestamps of command-line targets (or first makefile target) and executes preprocessing commands, but does not run the build)

  • /NOLOGO (Suppresses the NMAKE copyright message)

  • /ERRORREPORT (If nmake.exe fails at runtime, sends information to Microsoft about these internal errors)

Commands that Read from the Console

When GNU Make or NMAKE invokes a makefile target command, that process inherits the standard I/O streams of the Make process. It is then possible to invoke commands that expect input during a build, either from the terminal or passed into the Make standard input stream. For example, a makefile such as:

all:    @cat

could be invoked like this:

% echo hello | gmake

More commonly, a command in a makefile might prompt the user for some input, particularly if the command encounters an error or warning condition:

all:    @rm -i destroy
% gmakerm: remove regular file `destroy'? y

Neither of these constructs is generally recommended because systems that require runtime user input are tedious to invoke and extremely difficult to automate.

Because Accelerator runs commands on cluster hosts where processes do not inherit the I/O streams and console of the parent eMake, makefiles (such as the examples above) with commands expecting interactive input are not supported.

In the majority of cases, tools that prompt for console input contain options to disable interactive prompting and proceed automatically. For example, invoking “ rm ” without the “ i ” enables this behavior. For those that do not, explicitly feeding expected input (either from a file or directly by shell redirection or piping) will suffice. For example:

all:echo y | rm -i destroy

Finally, tools such as Expect (see ) that automate an interactive session can be used for commands that insist on reading from a console.

Transactional Command Output

eMake simulates a serial GNU Make or NMAKE build. Though eMake runs many commands in parallel, the command output (including text written to standard streams and changes to the file system, such as creating or updating files) appears serially in the original order without overlapping. This feature is called transactional output (or “serial order execution”) and is unique to Accelerator among parallel build systems. This feature ensures standard output streams and underlying file systems always reflect a consistent build execution state, regardless of how many jobs eMake is actually running concurrently on the cluster .

Transactional output is achieved by buffering the results of every command until the output from all preceding commands is written. Buffering means that while the output contents on the standard streams matches GNU Make or NMAKE exactly, the timing of its appearance might be a little unexpected. For example:

  • “Bursty” output – One of the first things you notice when running a build with Accelerator is that it appears to proceed in bursts, with many jobs finishing in quick succession followed by pauses. This type of output is normal during a highly parallel build because many later jobs might have completed and output is ready to be written as soon as longer, earlier jobs complete. The system remains busy, continuously running jobs throughout the build duration, even if the output appears to have paused momentarily.

  • Output follows job completion – GNU Make and NMAKE print commands they are executing before they are invoked. Because eMake is running many commands in parallel and buffering results to ensure transactional output, command-line text appears with the output from the command after the command has completed. For example, the last command printed on standard output is the job that just completed, not the one currently running.

  • Batch output – As a way to provide feedback to the user during a long-running execution, some commands might write to standard output continuously during their execution. Typically, these commands might print a series of ellipses or hash marks to indicate progress or might write status messages to standard error as they run. More commonly, a job might have several long-running commands separated with echo statements to report on progress during build execution:

For example, consider a rule that uses rsync to deploy output:

     @echo "Copying output into destination"
     rsync -vr $(OUTPUT) $(DESTINATION)
     @echo "done"

With GNU Make, users first see the Copying output echo, then the state information from rsync as it builds the file list, copies files, and finally, they see the done echo as the job completes.

With eMake, all output from this job step appears instantaneously in one burst when the job completes. By the time any output from echo or rsync is visible, the entire job has completed.

Stubbed Submake Output

Recursive Makes (also called submakes because they are invoked by a parent or top-level Make instance) are often used to partition a build into smaller modules. While submakes simplify the build system by allowing individual components to be built as autonomous units, they can introduce new problems because they fragment the dependency tree. Because the top-level Make does not coordinate with the submake—it is just another command—it is unable to track targets across Make instance boundaries. For a discussion of submake problems, see “Recursive Make Considered Harmful” by Peter Miller (

Submakes are particularly problematic for parallel builds because a build composed of separate Make instances is unable to control target serialization and concurrency between them. For example, consider a build divided into two phases: a libs submake that creates shared libraries followed by apps that builds executables that link against those libraries. A typical top-level makefile that controls this type of build might look like this:

all: makelibs makeapps
    $(MAKE) -C libs
    $(MAKE) -C apps

This type of makefile works fine for a serialized make, but running in parallel it can quickly become trouble:

  • If makelibs and makeapps run concurrently (as the makefile “all” rule implies), link steps in the apps Make instance might fail if they prematurely attempt to read libs generated libraries. Worse, they might link against existing, out-of-date library copies, producing incorrect output without error. This is a failure to correctly serialize dependent targets.

  • Alternatively, if apps is forced to wait until libs completes, even apps targets that do not depend on libs (for example, all the compilation steps, which are likely the bulk of the build) are serialized unnecessarily. This is a failure to maximize concurrency.

Submakes are often spawned indirectly from a script instead of by makefile commands.
    # 'do-libs' is a script that will invoke 'make'

which can make it difficult for a Make system to identify submake invocations, let alone attempt to ensure their correct, concurrent execution. These problems are exacerbated with distributed (cluster) parallel builds because each make invocation is running on a remote Agent.

Correct, highly concurrent parallel builds require a single, global dependency tree. Short of re-architecting a build with submakes into a single Make instance, this is very difficult to achieve with existing Make tools.

An ideal solution to parallelizing submakes has the following properties:

  • maximizes concurrency, even across make instances

  • serializes jobs that depend on output from other jobs

  • minimizes changes to the existing system (in particular, does not require eliminating submakes or prohibit their invocation from scripts)

Submake Stubs

The parallel submake problem is solved by introducing submake stubs. eMake dispatches all commands, regardless of tool (compiler, packager, submake, script, and so on) to cluster Agents. After the Agent executes a command, it sends the results (output to standard streams and exit status) back to eMake, which then sends the next job command.

If the command run by the Agent invokes eMake (either directly by the expanded $(MAKE) variable or indirectly through a script that calls emake ), a new top-level build is not started. Instead, an eMake process started on the Agent enters stub mode and it simply records details of its invocation (current working directory, command-line, environment, and so on) and immediately exits with status 0 (success) without writing output or reading any makefiles. The Agent then passes invocation details recorded by the stub back to the main eMake process on the host build machine, which starts a new Make instance and integrates its targets (which run in parallel on the cluster just like any other job) into the global dependency tree. Commands that follow a submake invocation are logically in a separate job serialized after the last job in the submake.

The following example illustrates this process:

    @echo "start libs"
    @$(MAKE) -C libs
    @echo "finish libs"

This example is diagrammed in steps as shown in the following illustrations.

  1. eMake determines that Makelibs target needs to be built.

  2. eMake connects to Electric Agent.

  3. eMake sends the first command, echo "start libs".

  4. The Agent invokes the command, echo "start libs", and captures the result.

  5. The Agent returns the result, "start libs", to eMake.

  6. eMake sends the second command, emake -f libs.mak.

  7. The Agent runs the second command, emake -f libs.mak.

  8. emake enters stub mode and records the current working directory, command, and environment, then exits with no output and status code 0 .

  9. Agent returns the stub mode result and a special message stating a new Make was invoked with recorded properties (eMake).

  1. eMake starts a new Make instance, reads the makefile, libs.mak, and integrates the file into the dependency tree.

  2. New jobs are created and run against the cluster to build all targets in the `libs.mak ` makefile.

  3. eMake splits the last command in the Makelibs target, echo "finish libs", into a continuation job that is defined to have a serial build order later than the last job in the submake, but there is no explicit dependency created that requires it to run later than any of the jobs in the submake. This means that it might run in parallel (or even before) the jobs in the submake, but if for some reason that is not safe, eMake will be able to detect a conflict and rerun the continuation job.

  1. eMake sends the last command, echo "finish libs".

  2. Agent runs the command, echo "finish libs", and captures the result.

  3. Agent returns the result, "finish libs", to eMake.

The eMake Stub solution addresses three basic parallel submake problems by:

  • Running parallel submakes in stub mode – Stubs finish instantaneously and discover jobs as quickly as possible so the build is as concurrent as possible.

  • Creating a single logical Make instance – New Make instances are started by the top-level eMake process only and their targets are integrated into the global dependency tree. eMake can then track dependencies across Make instances and use its conflict resolution technology to re-order jobs that might have run too soon.

  • Capturing submake invocations – By capturing submake invocations as they occur, the submake stub works with your makefiles and builds scripts for the majority of cases “as-is.” However, the stub introduces a behavior change that might require some makefile changes. See Submake Stub Compatibility.

Submake Stub Compatibility

Submake stubs allow your existing recursive builds to behave like a single logical Make instance to ensure fast, correct parallel builds. Stubs do introduce new behavior, however, that might appear as build failures. This section describes what constructs are not supported and what must be changed to make stubs compatible with submake stubs.

At this point, it is useful to revisit the relationship between commands and rules:

    echo "this is a command"
    echo "another command that includes a copy" ; \
    cp aa bb
    echo "so does this command" ; \
    cp bb cc
    cp cc dd

The rule above contains four commands: (1) echo, (2) echo-and-copy, (3) another echo-and-copy, and (4) a copy. Note how semicolons, backslashes, and new lines delimit (or continue) commands. The rule becomes a job when Make schedules it because it is needed to build the “all” target.

Submake stubs' most important features:

  • A submake stub never has any output and always exits successfully.

  • The Agent sends stub output back (if any) after each command.

  • Commands that follow a stub are invoked after the submake in the serial order.

Because a submake stub is really just a way of marking the Make invocation and does not actually do anything, you cannot rely on its output (stdout/stderr, exit status, or file system changes) in the same command .

In the following three examples, incompatible Accelerator commands are described and examples for fixing the incompatibility are provided.

Example 1: A command that reads submake file system changes

    $(MAKE) -C libs libcore.aa ; cp libs/libcore.aa /lib

In this example, a single command spawns a submake that builds libcore.a and then copies the new file into the /lib directory. If you run this rule as-is with Accelerator, the following error might appear:

cp: cannot stat ’libs/libcore.aa’: No such file or directorymake: *** [all] Error 1

The submake stub exited immediately and the cp begins execution right after it in the same command. eMake was not notified of the new Make instance yet, so no jobs to build libcore.a even exist. The cp fails because the expected file is not present.

An easy fix for this example is to remove the semicolon and make the cp a separate command:

    $(MAKE) -C libs libcore.aa
    cp libs/libcore.aa /lib

Now the cp is in a command after the submake stub sends its results back to the build machine. eMake forces the cp to wait until the submake jobs have completed, thus allowing the copy to succeed because the file is present. Note that this change has no effect on other Make variants so it will not prevent building with existing tools.

In general, Accelerator build failures that manifest themselves as apparent re-ordering or missing executions are usually because of commands reading the output of submake stubs. In most cases, the fix is simply to split the rule into multiple commands so the submake results are not read until after the submake completes.

Example 2: A command that reads submake stdout

makelibs:    $(MAKE) -C libs mkinstall >

The output is captured in a script that could be replayed later. Running this makefile with Accelerator always produces an empty because submake stubs do not write output. When eMake does invoke this Make instance, the output goes to standard output, as though no redirection was specified.

Commands that read from a Make stdout are relatively unusual. Those that do often read from a Make that does very little actual execution either because it is invoked with -n or because it runs a target that writes to stdout only. In these cases, it is not necessary to use a submake stub. The Make instance being spawned is small and fast, and running it directly on the Agent in its entirety (instead of returning to the host build machine and distributing individual jobs back to the cluster) does not significantly impact performance.

You can specify that an individual emake invocation on the Agent does not enter stub mode, but instead behaves like a local (non-cluster) Make simply by setting the EMAKE_BUILD_MODE environment variable for that instance:

    EMAKE_BUILD_MODE=local \
$(MAKE) -C libs mkinstall >

For Windows:

    set EMAKE_BUILD_MODE=local && $(MAKE) -C libs mkinstall >

eMake automatically uses local mode when the -n switch is specified.

Example 3: A command that reads submake exit status

makelibs:    $(MAKE) -C libs || echo "failure building libs"

This example is a common idiom for reporting errors. The || tells the shell to evaluate the second half of the expression only if Make exits with non-zero status. Again, because a submake stub always exits with 0, this clause will never be invoked Accelerator, even if it would be invoked with GNU Make. If you need this type of fail-over handling, consider post-processing the output log in the event of a build failure. Also see Annotation for more information.

Another common idiom in makefiles where exit status is read in loop constructs such as:

    for i in dir1 dir2 dir3 ; do \
        $(MAKE) -C $$i || exit 1;\

This is a single command: a “for” loop that spawns three submakes. The || exit 1 ` is present to prevent GNU Make from continuing to start the next submake if the current one fails. Without the `exit 1 clause, the command exit status is the exit status from the last submake, regardless of whether the preceding submakes succeeded or failed, or regardless of which error handling setting (for example, -i, -k ) was used in the Make. The || exit 1 idiom is used to force the submakes to better approximate the behavior of other Make targets, which stops the build on failure.

On first inspection, this looks like an unsupported construct for submake stubs because exit status is read from a stub. Accelerator never evaluates the || exit 1 ` because the stub always exits with status code 0. However, because the submakes really are reintegrated as targets in the top-level Make, a failure in one of them halts the build as intended. Explained another way, a submake loop is already treated as a series of independent targets, and the presence or absence of the GNU Make `|| exit 1 hint does not change this behavior. These constructs should be left as-is.

Hidden Targets

eMake differs from other Make variants in the way it searches for files needed by pattern rules (also called suffix or implicit rules) in a build.

  • At the beginning of each Make instance, eMake searches for matching files for all pattern rules before it runs any commands. After eMake has rules for every target that needs updating, it schedules the rules (creating jobs) and then runs those jobs in parallel for maximum concurrency.

  • Microsoft NMAKE and GNU Make match pattern rules as they run commands , interleaving execution and pattern search.

Because of the difference in the way eMake and NMAKE match pattern rules, NMAKE and eMake can produce different makefile output with hidden targets . A hidden target (also known as a “hidden dependency”) is a file that is:

  • created as an undeclared side-effect of updating another target

  • required by a pattern to build a rule

Consider the following makefile example:

all: bar.lib foo.obj
    touch bar.lib foo.c
    touch $@

Notice that foo.c is created as a side effect of updating the bar.lib target. Until bar.lib is updated, no rule is available to update foo.obj because nothing matches the .c.obj suffix rule.

NMAKE accepts this construct because it checks for foo.c existence before it attempts to update foo.obj. NMAKE produces the following result for this makefile:

touch bar.lib foo.c
touch foo.obj

eMake, however, performs the search for files that match the suffix rule once so it can schedule all jobs immediately and maximize concurrency. eMake will not notice the existence of foo.c by the time it attempts to update foo.obj, even if foo.c was created. eMake fails with:

NMAKE : fatal error U1073: don't know how to make 'foo.obj'Stop.

The fix is simply to identify foo.c as a product for updating the bar.lib target, so it is no longer a hidden target. For the example above, adding a line such as foo.c: bar.lib is sufficient for eMake to understand that .c.obj suffix rule matches the foo.obj target if bar.lib is built first. Adding this line is more accurate and has no effect on NMAKE.

GNU Make is similarly incompatible with eMake, but the incompatibility is sometimes masked by the GNU Make directory cache. GNU Make attempts to cache the directory contents on first access to improve performance. Unfortunately, because the time of first directory access can vary widely depending on which targets reference the directory and when they execute, GNU Make can appear to fail or succeed randomly in the presence of hidden targets.

For example, in this makefile, the file $(DIR)/foo.yy is a hidden target created as a side-effect of updating aa and needed by the pattern rule for foo.xx :

all: aa bb
     touch $(DIR)/foo.yybb: foo.xx
%.xx: $(DIR)/%.yy
    @echo $@

Depending on the value of DIR, this build might or might not work with GNU Make:

% mkdir sub; gmake DIR=sub
touch sub/foo.yy
% gmake DIR=.
touch ./foo.yy
gmake: *** No rule to make target ’foo.xx’, needed by ’bb’.  Stop.

eMake does not attempt to emulate this behavior. Instead, it consistently refuses to schedule foo.xx because it depends on a hidden target (just as it did in the NMAKE emulation mode in the earlier example). In this case, adding a single line declaring the target: $(DIR)/foo.yy: aa is sufficient to ensure it always matches the %.xx pattern rule.

If a build completes successfully with Microsoft NMAKE or GNU Make, but fails with “don’t know how to make <x> ” with eMake, look for rules that create <x> as a side-effect of updating another target. If <x> is required by a suffix rule also, it is a hidden target and needs to be declared as explicit output to be compatible with eMake.

There are many other reasons why hidden targets are problematic for all Make-based systems and why eliminating them is good practice in general. For more information, see:

In a limited number of cases, eMake might conclude that a matching pattern rule for an output target does not exist. This occurs because eMake’s strict string equality matching for prerequisites determines that the prerequisites are different (even though the paths refer to the same file) and that there is no rule to build it.

Wildcard Sort Order

A number of differences exist between GNU Make and eMake regarding the use of $(wildcard) and prerequisite wildcard sort order functions. When using the $(wildcard) function or using a wildcard in the rule prerequisite list, the resultant wildcard sort order might be different for GNU Make and eMake.

Different GNU Make versions are not consistent and exhibit permuted file lists. Even a GNU Make version using different system libraries versions will exhibit inconsistencies in the wildcard sort order.

No difference exists in the file list returned, other than the order. If the sort order is important, you might wrap $(wildcard) with $(sort).

For example:

$(sort $(wildcard *.foo))

Do not rely on the order of rule prerequisites generated with a wildcard. For example, using target: *.foo.

Relying on the order of *.foo can be dangerous for both GNU Make and eMake. Neither GNU Make nor eMake guarantees the order in which those prerequisites are executed.

Delayed Existence Checks

All Make variants process makefiles by looking for rules to build targets in the dependency tree. If no target rule is present, Make looks for a file on disk with the same name as the target. If this existence check fails, Make notes it has no rule to build the target.

eMake also exhibits this behavior, but for performance reasons it delays the existence check for a target without a makefile rule until just before that target is needed. The effect of this optimization is that eMake might run more commands to update targets (than GNU Make or NMAKE) before it discovers it has no rule to make a target.

For example, consider the following makefile:

all: nonexistent aa bb
    @echo $@
    @echo $@

GNU Make begins by looking for a rule for nonexistent and, when it does not find the rule, it does a file existence check. When that fails, GNU Make terminates immediately with:

make: *** No rule to make target ’nonexistent’, needed by ’all’.  Stop.

Similarly, NMAKE fails with:

NMAKE : fatal error U1073: don’t know how to make ’nonexistent’ Stop.

eMake delays the existence check for nonexistent until it is ready to run the all target. First, eMake finishes running commands to update the aa and bb prerequisites. eMake fails in the same way, but executes more targets first:

make: *** No rule to make target ’nonexistent’, needed by ’all’.  Stop.

Of course, when the existence check succeeds (as it does in any successful build), there is no behavioral difference between eMake and GNU Make or Microsoft NMAKE.

Multiple Remakes (GNU Make only)

GNU Make has an advanced feature called Makefile Remaking, which is documented in the GNU Manual, “How Makefiles are Remade,” and available at:

To quote from the GNU Make description:

“Sometimes makefiles can be remade from other files, such as RCS or SCCS files. If a makefile can be remade from other files, you probably want make to get an up-to-date version of the makefile to read in.

“To this end, after reading in all makefiles, make will consider each as a goal target and attempt to update it. If a makefile has a rule which says how to update it (found either in that very makefile or in another one) or if an implicit rule applies to it (see section Using Implicit Rules), it will be updated if necessary. After all makefiles have been checked, if any have actually been changed, make starts with a clean slate and reads all the makefiles over again. (It will also attempt to update each of them over again, but normally this will not change them again, since they are already up to date.)”

This feature can be very useful for writing makefiles that automatically generate and read dependency information with each build. However, this feature can cause GNU Make to loop infinitely if the rule to generate a makefile is always out-of-date:

    @echo $@
makefile: force
    @echo "# last updated: ’date’" >> $@

In practice, a well-written makefile will not have out-of-date rules that cause it to regenerate. The same problem, however, can occur when Make detects a clock skew—most commonly due to clock drift between the system running Make and the file server hosting the current directory. In this case, Make continues to loop until the rule to rebuild the makefile is no longer out-of-date.

In the example below, DIR1 and DIR2 are both part of the source tree:

-include $(DIR1)/foo.dd
    @echo $@
$(DIR1)/foo.dd: $(DIR2)/bar.dd
    touch $@

If two directories are served by different file servers and the clock on the system hosting DIR2 is slightly faster than DIR1, then even though foo.dd is updated after bar.dd, it might appear to be older. On remaking, GNU Make will again see foo.dd as out-of-date and restart, continuing until the drift is unnoticeable.

This problem is particularly troublesome on Solaris, where GNU Make timestamp checking has nanosecond resolution:

% make
touch dir2/bar.dd
touch dir1/foo.dd
gmake: *** Warning: File ’bar.dd’ has modification time in the future (2005-04-11 13:52:46.811724 > 2005-04-11 13:52:46.799573198)
touch dir1/foo.dd
touch dir1/foo.dd

eMake fully supports makefile remaking and can be configured to behave exactly as GNU Make. However, by default, to ensure builds do not loop unnecessarily while remaking, eMake limits the number of times it restarts a make instance to 10. If your build is looping unnecessarily, you might want to lower this value or disable remaking entirely by setting:


NMAKE Inline File Locations (Windows only)

NMAKE contains a feature to create inline files with temporary file names. For example, the following makefile creates a temporary inline file containing the word “pass” and then uses “type” to output it.

        type <<

With Microsoft NMAKE, the file is created in the directory where %TMP% points. eMake does not respect the %TMP% setting and creates the inline file in the rule working directory that needs the file.

How eMake Processes MAKEFLAGS

eMake uses the following process:

  1. Similar to GNU Make, eMake condenses no-value options into one block.

  2. When eMake encounters an option with a value, it does what GNU Make does, it appends the value and starts the next option with its own -

  3. Certain options are ignored/not created. This changes the layout of the options in MAKEFLAGS (for example -j, -l ).

  4. eMake-specific options are not added to MAKEFLAGS, but are handled through EMAKEFLAGS.

  5. Passing down environment variables as TEST=test renders the same result as in GNU Make (an extra - at the end, followed by the variable=value ).

  6. On Windows, eMake prepends the --unix or --win32 flag explicitly.