Simulating builds

5 minute read

CloudBees Build Acceleration includes moxie, a tool which can simulate builds. Moxie allows users to run a realistic build without any external dependencies, providing the opportunity to test your eMake environment and also to see the speedups eMake can achieve with various options (for example, JobCache or parse avoidance). Moxie also validates these simulated builds with the use of an accumulated hash.

It accomplishes this by invoking itself when a simulated build is running, performing the relevant I/O (for example, reading files and then writing files), and sleeping. Because of this, it can simulate any step in a toolchain in a build—for example, a compiler, a linker, or an archiver.

Since moxie only performs I/O and sleeps, it uses very little CPU, but in all other respects acts like a discrete build step. Because of this, the speedups that eMake can achieve with simulated builds are precisely analogous to the speedups that would occur with a real build—what makes a simulated build faster is exactly what makes a real build faster.

The amount of time that simulated builds take is designed to be consistent across runs and even across different platforms, though naturally the speedups achieved with eMake will vary depending on the number of Agents in your cluster, current cluster usage, and so on.

Simulated build environments

The moxie tool is also used to create simulated build environments. It does this by reading a build specification file with a .mox extension, which describes the directories and files to create and their contents. The default build specifications are installed in /opt/ecloud/i686_Linux/moxie-specs (C:\ECloud\i686_win32\moxie-specs on Windows). To create a build environment, simply invoke moxie with the path to a .mox file as its argument. Moxie is in the same binaries directory as eMake.

To run a simulated build, create the build environment by running moxie with a .mox argument, change to the directory it creates, and then invoke GNU Make and/or eMake. The example shows acceleration on UNIX.

% moxie /opt/ecloud/i686_Linux/moxie-specs/sample.mox Created sample build under the directory "sample" Change to that directory and run [e]make. % cd sample % time make moxie -m configure -i -o common.h moxie -m compile -i alpha.c alpha.h echo.h india.h oscar.h uniform.h common.h -o alpha.o moxie -m compile -i bravo.c bravo.h alpha.h echo.h india.h oscar.h uniform.h common.h -o bravo.o moxie -m compile -i charlie.c charlie.h alpha.h echo.h india.h oscar.h uniform.h common.h -o charlie.o ... moxie -m link -i alpha.o bravo.o charlie.o delta.o echo.o foxtrot.o golf.o hotel.o india.o juliet.o kilo.o lima.o mike.o november.o oscar.o papa.o quebec.o romeo.o sierra.o tango.o uniform.o victor.o whiskey.o xray.o yankee.o zulu.o generated.o main.o -o test.out -H 3246985f199ba295 moxie -m archive -i test.out libtest.so alpha.h bravo.h charlie.h delta.h echo.h foxtrot.h golf.h hotel.h india.h juliet.h kilo.h lima.h mike.h november.h oscar.h papa.h quebec.h romeo.h sierra.h tango.h uniform.h victor.h whiskey.h xray.h yankee.h zulu.h common.h -o test.zip -H 00482e54fdc2c8b2 make 0.04s user 0.02s system 0% cpu 1:02.81 total % make clean rm -f common.h generated.c generated.o alpha.d bravo.d charlie.d delta.d echo.d foxtrot.d golf.d hotel.d india.d juliet.d kilo.d lima.d mike.d november.d oscar.d papa.d quebec.d romeo.d sierra.d tango.d uniform.d victor.d whiskey.d xray.d yankee.d zulu.d alpha.o bravo.o charlie.o delta.o echo.o foxtrot.o golf.o hotel.o india.o juliet.o kilo.o lima.o mike.o november.o oscar.o papa.o quebec.o romeo.o sierra.o tango.o uniform.o victor.o whiskey.o xray.o yankee.o zulu.o main.o libtest.a libtest.so.nostrip libtest.so test.out test.zip % time emake --emake-cm=localhost Running Build - ID: 4455 Elapsed Time: 0m25.226s Percentage Complete: 100% Conflicts: 0 Agents In Use: 2 Jobs: Runnable: 0 Completed: 45 Terminated: 45 Total: 45 Finished build: 4455 Duration: 0:25 (m:s) Cluster availability: 100% emake 1.55s user 0.62s system 8% cpu 25.291 total
By default eMake shows progress and status information on the console when running on Linux, with the traditional build log captured in the file emake.out. To have the build log written to the console instead, add --emake-show-progress=0 to the eMake command-line arguments.

If moxie is not in your PATH or you wish to provide extra options to moxie when it is invoked by the simulated build, use the MOXIE variable:

% [e]make MOXIE=/path/to/moxie % [e]make MOXIE="moxie <options>"

Default build specifications

Three simulated build specifications are installed in /opt/ecloud/i686_Linux/moxie-specs (C:\ECloud\i686_win32\moxie-specs on Windows) by default:

  • sample.mox simulates a relatively straightforward but typical build involving a configure-like step, compiling a series of source files to object files, linking those object files into both a static library and a dynamic library, stripping the dynamic library, linking the objects into an executable, and then creating an archive from the executable and header files. With GNU Make, it will build in about a minute.

  • alphabets.mox simulates a larger build involving a hierarchy of directories, each of which contains a series of source files which are compiled and then linked to a library. These libraries are then linked together with a main object file into an executable. It will build in about five minutes with GNU Make.

  • world.mox simulates a very large build involving a much larger hierarchy of directories containing source files. Again, each of these are compiled and then linked into a library. The resulting libraries are all put into an archive. About an hour is required to run this build with GNU Make.

Timing

The following table shows typical build times and their corresponding speedups against the baseline for all three default build specifications with the following builds:

  • GNU Make (this is used as the baseline).

  • GNU Make with -j6 (that is, with 6 concurrent jobs).

  • eMake with the default options on a Cluster Manager with 6 agents.

  • eMake as above but with JobCache enabled (--emake-jobcache=all). Note that this involves two consecutive builds: first to train (build the cache), and a second time to use the cache.

Specification GNU Make GNU Make with -j6 eMake eMake with JobCache

sample.mox

1:03

0:25 (2.5x)

0:25 (2.5x)

0:21 (3.0x)

alphabets.mox

5:49

2:09 (2.71x)

1:08 (5.1x)

0:22 (15.9x)

world.mox

1:08:46

20:21 (3.38x)

11:47 (5.84x)

1:03 (65.5x)

Note that the speedups are considerably greater for the larger, more hierarchical builds, and using JobCache makes them greater still. Using a cluster with more Agents would increase the speedup even more.

Command line options

The full list of command line options will be shown with the -h/--help option.

Short name Long name Description

-h

--help

Print usage and exit. This is also performed if moxie is invoked with no arguments.

-V

--version

Print the version of moxie and exit.

-q

--quiet

Disable all logging (including that of warnings and errors).

n/a

--default

Log only warnings and errors (default).

-b

--brief

Log sleeps, warnings and errors.

-v

--verbose

Log inputs, sleeps, outputs (with hashes), warnings and errors.

n/a

--debug

Log all of the above as well as additional debugging output.

n/a

--timestamps

Log timestamps.

n/a

--no-timestamps

Do not log timestamps (default).

n/a

--levels

Log logging levels. For example, (INFO) (default).

n/a

--no-levels

Do not log logging levels.

n/a

--components

Log components. For example, [sleep] (default).

n/a

--no-components

Do not log components.

n/a

--localtime

Use local time for timestamps.

n/a

--gmtime

Use GMT for timestamps.

-I

--instantaneous

Do not sleep. This is useful for build validation where the build duration is not of immediate interest.

--

n/a

Ignore all following command line arguments (note this is slightly different from GNU option standards).

Exit codes

Exit code Description

0

Success.

2

An argument to a command line option was invalid (in the wrong format).

3

A given command line option was unrecognized.

4

The provided mode name was unrecognized.

5

An incorrect number of colon-separated arguments was provided to an option.

6

A build specification contained an unknown instruction.

7

An attempt to create a directory or write a file from a build specification failed.

8

An invalid character was encountered in a BINARY build specification instruction.

9

A build specification file could not be open or read.

10

A line containing a build specification instruction was too long.

20

The provided internal build specification name was unrecognized.

30

The given command is currently unsupported.

40

There were too many commands.

99

The provided hash to validate (with the -H/--hash option) did not match.

100

The assertion failed.

126

An allocation failed because the machine is out of memory.