CloudBees Build Acceleration includes moxie, a tool which can simulate builds. Moxie allows users to run a realistic build without any external dependencies, providing the opportunity to test your eMake environment and also to see the speedups eMake can achieve with various options (for example, JobCache or parse avoidance). Moxie also validates these simulated builds with the use of an accumulated hash.
It accomplishes this by invoking itself when a simulated build is running, performing the relevant I/O (for example, reading files and then writing files), and sleeping. Because of this, it can simulate any step in a toolchain in a build—for example, a compiler, a linker, or an archiver.
Since moxie only performs I/O and sleeps, it uses very little CPU, but in all other respects acts like a discrete build step. Because of this, the speedups that eMake can achieve with simulated builds are precisely analogous to the speedups that would occur with a real build—what makes a simulated build faster is exactly what makes a real build faster.
The amount of time that simulated builds take is designed to be consistent across runs and even across different platforms, though naturally the speedups achieved with eMake will vary depending on the number of Agents in your cluster, current cluster usage, and so on.
Simulated build environments
The moxie tool is also used to create simulated build environments. It does this by reading a build specification file with a .mox extension, which describes the directories and files to create and their contents. The default build specifications are installed in /opt/ecloud/i686_Linux/moxie-specs (C:\ECloud\i686_win32\moxie-specs on Windows). To create a build environment, simply invoke moxie with the path to a .mox file as its argument. Moxie is in the same binaries directory as eMake.
To run a simulated build, create the build environment by running moxie with a .mox argument, change to the directory it creates, and then invoke GNU Make and/or eMake. The example shows acceleration on UNIX.
% moxie /opt/ecloud/i686_Linux/moxie-specs/sample.mox Created sample build under the directory "sample" Change to that directory and run [e]make. % cd sample % time make moxie -m configure -i -o common.h moxie -m compile -i alpha.c alpha.h echo.h india.h oscar.h uniform.h common.h -o alpha.o moxie -m compile -i bravo.c bravo.h alpha.h echo.h india.h oscar.h uniform.h common.h -o bravo.o moxie -m compile -i charlie.c charlie.h alpha.h echo.h india.h oscar.h uniform.h common.h -o charlie.o ... moxie -m link -i alpha.o bravo.o charlie.o delta.o echo.o foxtrot.o golf.o hotel.o india.o juliet.o kilo.o lima.o mike.o november.o oscar.o papa.o quebec.o romeo.o sierra.o tango.o uniform.o victor.o whiskey.o xray.o yankee.o zulu.o generated.o main.o -o test.out -H 3246985f199ba295 moxie -m archive -i test.out libtest.so alpha.h bravo.h charlie.h delta.h echo.h foxtrot.h golf.h hotel.h india.h juliet.h kilo.h lima.h mike.h november.h oscar.h papa.h quebec.h romeo.h sierra.h tango.h uniform.h victor.h whiskey.h xray.h yankee.h zulu.h common.h -o test.zip -H 00482e54fdc2c8b2 make 0.04s user 0.02s system 0% cpu 1:02.81 total % make clean rm -f common.h generated.c generated.o alpha.d bravo.d charlie.d delta.d echo.d foxtrot.d golf.d hotel.d india.d juliet.d kilo.d lima.d mike.d november.d oscar.d papa.d quebec.d romeo.d sierra.d tango.d uniform.d victor.d whiskey.d xray.d yankee.d zulu.d alpha.o bravo.o charlie.o delta.o echo.o foxtrot.o golf.o hotel.o india.o juliet.o kilo.o lima.o mike.o november.o oscar.o papa.o quebec.o romeo.o sierra.o tango.o uniform.o victor.o whiskey.o xray.o yankee.o zulu.o main.o libtest.a libtest.so.nostrip libtest.so test.out test.zip % time emake --emake-cm=localhost Running Build - ID: 4455 Elapsed Time: 0m25.226s Percentage Complete: 100% Conflicts: 0 Agents In Use: 2 Jobs: Runnable: 0 Completed: 45 Terminated: 45 Total: 45 Finished build: 4455 Duration: 0:25 (m:s) Cluster availability: 100% emake 1.55s user 0.62s system 8% cpu 25.291 total
By default eMake shows progress and status information on the console when running on Linux, with the traditional build log captured in the file emake.out. To have the build log written to the console instead, add --emake-show-progress=0 to the eMake command-line arguments.
|
If moxie is not in your PATH or you wish to provide extra options to moxie when it is invoked by the simulated build, use the MOXIE variable:
% [e]make MOXIE=/path/to/moxie % [e]make MOXIE="moxie <options>"
Default build specifications
Three simulated build specifications are installed in /opt/ecloud/i686_Linux/moxie-specs (C:\ECloud\i686_win32\moxie-specs on Windows) by default:
-
sample.moxsimulates a relatively straightforward but typical build involving a configure-like step, compiling a series of source files to object files, linking those object files into both a static library and a dynamic library, stripping the dynamic library, linking the objects into an executable, and then creating an archive from the executable and header files. With GNU Make, it will build in about a minute. -
alphabets.moxsimulates a larger build involving a hierarchy of directories, each of which contains a series of source files which are compiled and then linked to a library. These libraries are then linked together with a main object file into an executable. It will build in about five minutes with GNU Make. -
world.moxsimulates a very large build involving a much larger hierarchy of directories containing source files. Again, each of these are compiled and then linked into a library. The resulting libraries are all put into an archive. About an hour is required to run this build with GNU Make.
Timing
The following table shows typical build times and their corresponding speedups against the baseline for all three default build specifications with the following builds:
-
GNU Make (this is used as the baseline).
-
GNU Make with
-j6(that is, with 6 concurrent jobs). -
eMake with the default options on a Cluster Manager with 6 agents.
-
eMake as above but with JobCache enabled (
--emake-jobcache=all). Note that this involves two consecutive builds: first to train (build the cache), and a second time to use the cache.
| Specification | GNU Make | GNU Make with -j6 | eMake | eMake with JobCache |
|---|---|---|---|---|
sample.mox |
1:03 |
0:25 (2.5x) |
0:25 (2.5x) |
0:21 (3.0x) |
alphabets.mox |
5:49 |
2:09 (2.71x) |
1:08 (5.1x) |
0:22 (15.9x) |
world.mox |
1:08:46 |
20:21 (3.38x) |
11:47 (5.84x) |
1:03 (65.5x) |
Note that the speedups are considerably greater for the larger, more hierarchical builds, and using JobCache makes them greater still. Using a cluster with more Agents would increase the speedup even more.
Command line options
The full list of command line options will be shown with the -h/--help option.
| Short name | Long name | Description |
|---|---|---|
|
|
Print usage and exit. This is also performed if moxie is invoked with no arguments. |
|
|
Print the version of moxie and exit. |
|
|
Disable all logging (including that of warnings and errors). |
n/a |
|
Log only warnings and errors (default). |
|
|
Log sleeps, warnings and errors. |
|
|
Log inputs, sleeps, outputs (with hashes), warnings and errors. |
n/a |
|
Log all of the above as well as additional debugging output. |
n/a |
|
Log timestamps. |
n/a |
|
Do not log timestamps (default). |
n/a |
|
Log logging levels. For example, |
n/a |
|
Do not log logging levels. |
n/a |
|
Log components. For example, |
n/a |
|
Do not log components. |
n/a |
|
Use local time for timestamps. |
n/a |
|
Use GMT for timestamps. |
|
|
Do not sleep. This is useful for build validation where the build duration is not of immediate interest. |
|
n/a |
Ignore all following command line arguments (note this is slightly different from GNU option standards). |
Exit codes
| Exit code | Description |
|---|---|
0 |
Success. |
2 |
An argument to a command line option was invalid (in the wrong format). |
3 |
A given command line option was unrecognized. |
4 |
The provided mode name was unrecognized. |
5 |
An incorrect number of colon-separated arguments was provided to an option. |
6 |
A build specification contained an unknown instruction. |
7 |
An attempt to create a directory or write a file from a build specification failed. |
8 |
An invalid character was encountered in a |
9 |
A build specification file could not be open or read. |
10 |
A line containing a build specification instruction was too long. |
20 |
The provided internal build specification name was unrecognized. |
30 |
The given command is currently unsupported. |
40 |
There were too many commands. |
99 |
The provided hash to validate (with the |
100 |
The assertion failed. |
126 |
An allocation failed because the machine is out of memory. |