Serializing All Make Instance Jobs

2 minute read

Normally, eMake runs all jobs in a Make instance in parallel on multiple distinct Agents. For most builds, this process ensures the best possible performance.

In some cases, however, all jobs in a Make instance are interdependent and must be run serially, for example, a set of jobs updating a shared file. In particular, Microsoft Visual C/C++ compilers exhibit this behavior when they create a common program database (PDB) file to store symbol and debug information to all object files.

Example

The following makefile invokes cl with the /Zi flag to specify the program database file be created with type and symbolic debugging information:

all:
    $(MAKE) my.exe
       $(MAKE) other.exe

my.exe: a.obj b.obj c.obj my.obj
        cl /nologo /Zi $^ /Fe'my.exe'

%.obj: %.c

        cl /nologo /Zi /c /Fo$@ $^

In this build, the a.obj, b.obj, c.obj ` and `my.obj jobs are implicitly serialized because they all write to the shared PDB file (by default, vc70.pdb ). In this case, jobs run in parallel, and running them on separate Agents only introduces unnecessary network overhead. This job type needs to run serially so it can correctly update the PDB file.

The eMake special directive, #pragma allserial used in the makefile, allows you to disable a parallel build in a Make instance and run the job serially on a single Agent. By inserting the #pragma allserial directive at the beginning of a line anywhere in the makefile, the directive specifies that all jobs in that make instance be serialized. This process maximizes network and file cache efficiency.

In the example above, by prefixing the %.obj ` pattern rule with the `#pragma allserial directive:

#pragma allserial
%.obj: %.c    cl /nologo /Zi /c /Fo$@ $^

eMake runs compiles and links for the my.exe Make instance in serial on the same Agent.

Splitting PDBs Using hashstr.exe

The hashstr.exe utility creates a hash of the file name given a modulus (maximum number of PDBs that will be produced). A given file must always produce the same PDB or history would constantly change. The hash should only include the file name and not its full path. Precompiled headers (PCHs) must be turned off.

Example

Usage: hashstr "mystring" [modulus]

Where mystring is the string from which to generate the hash value, and modulus is the number of hash bins you want to use.

You can add this to a pattern rule for builds that suffer from performance degradation due to PDB serialization, with something similar to the following:

%.o: %.c$(CC) /c $(cflags) $(PCH_USE_FLAGS) $(cvars) $(cplus_flags) $(LOCAL_INCLUDE)
$(PCB_INCLUDE) $< /Fo$@ /Fd$(shell ${path-to-hashstr}/hashstr.exe "$@"
${hashstr-modulus}).pdb