Stubbed Submake output

Recursive Makes (also called submakes because they are invoked by a parent or top-level Make instance) are often used to partition a build into smaller modules. While submakes simplify the build system by allowing individual components to be built as autonomous units, they can introduce new problems because they fragment the dependency tree. Because the top-level Make does not coordinate with the submake—it is just another command—it is unable to track targets across Make instance boundaries. For a discussion of submake problems, see “Recursive Make Considered Harmful” by Peter Miller (https://www.scribd.com/document/15677619/Recursive-Make-Considered-Harmful).

Submakes are particularly problematic for parallel builds because a build composed of separate Make instances is unable to control target serialization and concurrency between them. For example, consider a build divided into two phases: a libs submake that creates shared libraries followed by apps that builds executables that link against those libraries. A typical top-level makefile that controls this type of build might look like this:

all: makelibs makeapps
makelibs:
    $(MAKE) -C libs
makeapps:
    $(MAKE) -C apps▼

This type of makefile works fine for a serialized make, but running in parallel it can quickly become trouble:

If makelibs and makeapps run concurrently (as the makefile “all” rule implies), link steps in the apps Make instance might fail if they prematurely attempt to read libs generated libraries. Worse, they might link against existing, out-of-date library copies, producing incorrect output without error. This is a failure to correctly serialize dependent targets.
Alternatively, if apps is forced to wait until libs completes, even apps targets that do not depend on libs (for example, all the compilation steps, which are likely the bulk of the build) are serialized unnecessarily. This is a failure to maximize concurrency.

Submakes are often spawned indirectly from a script instead of by makefile commands.

makelibs:
    # 'do-libs' is a script that will invoke 'make'
    do-libs▼

which can make it difficult for a Make system to identify submake invocations, let alone attempt to ensure their correct, concurrent execution. These problems are exacerbated with distributed (cluster) parallel builds because each make invocation is running on a remote Agent.

Correct, highly concurrent parallel builds require a single, global dependency tree. Short of re-architecting a build with submakes into a single Make instance, this is very difficult to achieve with existing Make tools.

An ideal solution to parallelizing submakes has the following properties:

maximizes concurrency, even across make instances
serializes jobs that depend on output from other jobs
minimizes changes to the existing system (in particular, does not require eliminating submakes or prohibit their invocation from scripts)

Submake stubs

The parallel submake problem is solved by introducing submake stubs. eMake dispatches all commands, regardless of tool (compiler, packager, submake, script, and so on) to cluster Agents. After the Agent executes a command, it sends the results (output to standard streams and exit status) back to eMake, which then sends the next job command.

If the command run by the Agent invokes eMake (either directly by the expanded $(MAKE) variable or indirectly through a script that calls emake ), a new top-level build is not started. Instead, an eMake process started on the Agent enters stub mode and it simply records details of its invocation (current working directory, command-line, environment, and so on) and immediately exits with status 0 (success) without writing output or reading any makefiles. The Agent then passes invocation details recorded by the stub back to the main eMake process on the host build machine, which starts a new Make instance and integrates its targets (which run in parallel on the cluster just like any other job) into the global dependency tree. Commands that follow a submake invocation are logically in a separate job serialized after the last job in the submake.

The following example illustrates this process:

...
Makelibs:
    @echo "start libs"
    @$(MAKE) -C libs
    @echo "finish libs"▼

This example is diagrammed in steps as shown in the following illustrations.

eMake determines that Makelibs target needs to be built.
eMake connects to Electric Agent.
eMake sends the first command, echo "start libs".
The Agent invokes the command, echo "start libs", and captures the result.
The Agent returns the result, "start libs", to eMake.
eMake sends the second command, emake -f libs.mak.
The Agent runs the second command, emake -f libs.mak.
emake enters stub mode and records the current working directory, command, and environment, then exits with no output and status code 0 .
Agent returns the stub mode result and a special message stating a new Make was invoked with recorded properties (eMake).

eMake starts a new Make instance, reads the makefile, libs.mak, and integrates the file into the dependency tree.
New jobs are created and run against the cluster to build all targets in the `libs.mak ` makefile.
eMake splits the last command in the Makelibs target, echo "finish libs", into a continuation job that is defined to have a serial build order later than the last job in the submake, but there is no explicit dependency created that requires it to run later than any of the jobs in the submake. This means that it might run in parallel (or even before) the jobs in the submake, but if for some reason that is not safe, eMake will be able to detect a conflict and rerun the continuation job.

eMake sends the last command, echo "finish libs".
Agent runs the command, echo "finish libs", and captures the result.
Agent returns the result, "finish libs", to eMake.

The eMake Stub solution addresses three basic parallel submake problems by:

Running parallel submakes in stub mode – Stubs finish instantaneously and discover jobs as quickly as possible so the build is as concurrent as possible.
Creating a single logical Make instance – New Make instances are started by the top-level eMake process only and their targets are integrated into the global dependency tree. eMake can then track dependencies across Make instances and use its conflict resolution technology to re-order jobs that might have run too soon.
Capturing submake invocations – By capturing submake invocations as they occur, the submake stub works with your makefiles and builds scripts for the majority of cases “as-is.” However, the stub introduces a behavior change that might require some makefile changes. See Submake stub compatibility.

Submake stub compatibility

Submake stubs allow your existing recursive builds to behave like a single logical Make instance to ensure fast, correct parallel builds. Stubs do introduce new behavior, however, that might appear as build failures. This section describes what constructs are not supported and what must be changed to make stubs compatible with submake stubs.

At this point, it is useful to revisit the relationship between commands and rules:

all:
    echo "this is a command"
    echo "another command that includes a copy" ; \
    cp aa bb
    echo "so does this command" ; \
    cp bb cc
    cp cc dd▼

The rule above contains four commands: (1) echo, (2) echo-and-copy, (3) another echo-and-copy, and (4) a copy. Note how semicolons, backslashes, and new lines delimit (or continue) commands. The rule becomes a job when Make schedules it because it is needed to build the “all” target.

Submake stubs' most important features:

A submake stub never has any output and always exits successfully.
The Agent sends stub output back (if any) after each command.
Commands that follow a stub are invoked after the submake in the serial order.

Because a submake stub is really just a way of marking the Make invocation and does not actually do anything, you cannot rely on its output (stdout/stderr, exit status, or file system changes) in the same command .

In the following three examples, incompatible CloudBees Build Acceleration commands are described and examples for fixing the incompatibility are provided.

Example 1: A command that reads submake file system changes

makelibs:
    $(MAKE) -C libs libcore.aa ; cp libs/libcore.aa /lib▼

In this example, a single command spawns a submake that builds libcore.a and then copies the new file into the /lib directory. If you run this rule as-is with CloudBees Build Acceleration, the following error might appear:

cp: cannot stat ’libs/libcore.aa’: No such file or directorymake: *** [all] Error 1▼

The submake stub exited immediately and the cp begins execution right after it in the same command. eMake was not notified of the new Make instance yet, so no jobs to build libcore.a even exist. The cp fails because the expected file is not present.

An easy fix for this example is to remove the semicolon and make the cp a separate command:

makelibs:
    $(MAKE) -C libs libcore.aa
    cp libs/libcore.aa /lib▼

Now the cp is in a command after the submake stub sends its results back to the build machine. eMake forces the cp to wait until the submake jobs have completed, thus allowing the copy to succeed because the file is present. Note that this change has no effect on other Make variants so it will not prevent building with existing tools.

In general, CloudBees Build Acceleration build failures that manifest themselves as apparent re-ordering or missing executions are usually because of commands reading the output of submake stubs. In most cases, the fix is simply to split the rule into multiple commands so the submake results are not read until after the submake completes.

Example 2: A command that reads submake stdout

makelibs:    $(MAKE) -C libs mkinstall > installer.sh▼

The output is captured in a script that could be replayed later. Running this makefile with CloudBees Build Acceleration always produces an empty installer.sh because submake stubs do not write output. When eMake does invoke this Make instance, the output goes to standard output, as though no redirection was specified.

Commands that read from a Make stdout are relatively unusual. Those that do often read from a Make that does very little actual execution either because it is invoked with -n or because it runs a target that writes to stdout only. In these cases, it is not necessary to use a submake stub. The Make instance being spawned is small and fast, and running it directly on the Agent in its entirety (instead of returning to the host build machine and distributing individual jobs back to the cluster) does not significantly impact performance.

You can specify that an individual emake invocation on the Agent does not enter stub mode, but instead behaves like a local (non-cluster) Make simply by setting the EMAKE_BUILD_MODE environment variable for that instance:

makelibs:
    EMAKE_BUILD_MODE=local \
$(MAKE) -C libs mkinstall >
    installer.sh▼

For Windows:

makelibs:
    set EMAKE_BUILD_MODE=local && $(MAKE) -C libs mkinstall >
        installer▼

eMake automatically uses local mode when the -n switch is specified.

Example 3: A command that reads submake exit status

makelibs:    $(MAKE) -C libs || echo "failure building libs"▼

This example is a common idiom for reporting errors. The || tells the shell to evaluate the second half of the expression only if Make exits with non-zero status. Again, because a submake stub always exits with 0, this clause will never be invoked with CloudBees Build Acceleration, even if it would be invoked with GNU Make. If you need this type of fail-over handling, consider post-processing the output lsign in the event of a build failure. Also see " Annotation " for more information.

Another common idiom in makefiles where exit status is read in loop constructs such as:

all:
    for i in dir1 dir2 dir3 ; do \
        $(MAKE) -C $$i || exit 1;\
    done▼

This is a single command: a “for” loop that spawns three submakes. The || exit 1 ` is present to prevent GNU Make from continuing to start the next submake if the current one fails. Without the `exit 1 clause, the command exit status is the exit status from the last submake, regardless of whether the preceding submakes succeeded or failed, or regardless of which error handling setting (for example, -i, -k ) was used in the Make. The || exit 1 idiom is used to force the submakes to better approximate the behavior of other Make targets, which stops the build on failure.

On first inspection, this looks like an unsupported construct for submake stubs because exit status is read from a stub. CloudBees Build Acceleration never evaluates the || exit 1 ` because the stub always exits with status code 0. However, because the submakes really are reintegrated as targets in the top-level Make, a failure in one of them halts the build as intended. Explained another way, a submake loop is already treated as a series of independent targets, and the presence or absence of the GNU Make `|| exit 1 hint does not change this behavior. These constructs should be left as-is.