CloudBees Build Acceleration includes a parse avoidance feature that can almost eliminate makefile parse time. By caching and reusing parse result files, CloudBees Build Acceleration can speed up both full builds and incremental builds.
Parse avoidance works when emulating GNU Make with clusters or local agents.
Parse avoidance uses the concept of cache "slots." When parse avoidance is enabled, eMake maintains a slot for each combination of non-ignored command line options, non-ignored environment variable assignments, and current working directory. (Ignored arguments and environment variables are listed below.) A slot can be empty or hold a previously cached result. If the appropriate slot holds an up-to-date result, parsing is avoided.
A cached result becomes obsolete if eMake detects file system changes that might have caused a different result (with the same command line options, environment variable assignments, and current working directory). Such file system changes include any file read by the parse job, which means all makefiles, all programs invoked by $(shell) during the parse (as opposed to during rule execution), the files they read, and so on.
Console output
When a cached parse result is reused, eMake replays the file system modifications made during the parse job and any standard output or standard error that the original parse job produced. For example, if the makefile includes
VARIABLE1 := $(shell echo Text > myfile)VARIABLE2 := $(warning Warning: updated myfile)then when using a cached parse result, eMake creates "myfile" and prints "Warning: updated myfile".
Enabling parse avoidance
You must first run a "learning" build with the parse avoidance feature enabled. To enable parse avoidance, set the following: --emake-parse-avoidance=1
For the learning build, (because the parse cache is empty), the argument only saves a new result to the cache. For subsequent builds, the argument enables the reuse of cached parse results and saves a new result to the cache as appropriate. If you do not specify --emake-parse-avoidance=1, then the parse avoidance cache is not accessed at all.
You should also disable generated dependencies (either by modifying your makefiles or by using --emake-suppress-include ; see below)
|
The following table describes parse avoidance-related options.
| eMake Option | Description | ||
|---|---|---|---|
--emake-assetdir=< path > | Use the specified directory for cached parse results. The default directory is named .emake. (This option also determines the location of the saved dependency information for the dependency optimization feature and the cache location for JobCache .) | ||
--emake-parse-avoidance= < 0/1 > | Avoid parsing makefiles when prior parse results remain up-to-date and cache new parse results when appropriate. | ||
--emake-parse-avoidance-ignore-env=< var > | Ignore the named environment variable when searching for applicable cached parse results. To ignore more than one variable, use this option multiple times. | ||
--emake-parse-avoidance-ignore-path=< path > | Ignore this file or directory when checking whether cached parse results are up-to-date. Append % for prefix matching (the % must be the last character). To ignore more than one path or prefix, use this option multiple times. Incorrect placement of Correct: Incorrect: | ||
--emake-suppress-include= < pattern > | Skip matching makefile includes (such as generated dependencies). Generally, you should not suppress makefile includes unless they are generated dependency files, and you have enabled automatic dependencies as an alternative way of handling dependencies.
|
If a file is read during a parse but changes before eMake attempts to reuse that parse’s results, the cached parse result is normally considered to be obsolete. You can, however, temporarily override this decision with --emake-parse-avoidance-ignore-path.
eMake permanently ignores the effect on a given parse result of certain special files that might have existed when it was created, in all cases using the names they would have had at that time:
-
Anything in ".emake" or whatever alternative directory you specified using
--emake-assetdir -
Client-side eMake debug log (as specified by
--emake-logfile) -
Annotation file (as specified by
--emake-annofile) -
History file (as specified by
--emake-historyfile)
You can use a pragma to instruct parse avoidance to detect the dependence of a parse result upon path wildcard match results. This pragma makes the parse cache sensitive to the existence (or nonexistence) of specific files in directories that it reads as well as in new subdirectories thereof. (eMake does not detect matching files in just any new subdirectory, but only those that are subdirectories of directories read in the original job and their subdirectories, recursively.) You can specify more than one glob pattern, as in the following pragma: For Android-specific information about parse avoidance and other Android best practices, see Optimizing Android Build Performance. === Parse avoidance example
The following parse avoidance example provides command line arguments for Android 4.1.1.
== Deleting the cache To delete the cache, delete For example: == Moving Your workspace If you want to move your workspace, make sure that the new eMake roots correspond to the old eMake roots. Also, because the asset directory defaults to eMake looks for eMake roots in the values of Make variables. When eMake replays a cached parse result in a new workspace, it replaces the old eMake roots in the values of those variables with the new eMake roots. This policy works well as long as no confusion exists between eMake roots and other text in those variable values. For example, if the value of a Make variable is == Parse avoidance limitations
== Troubleshooting === Enabling parse avoidance debug logging To help troubleshoot performance issues, you should enable parse avoidance debug logging. To do so, include a capital letter “P” in the argument of the === Debug log example You can look in "P" debug logging for an explanation of why a cached result was found to be obsolete: Notice that " === Using key files for debugging To discover why a new cache slot was used, look in "P" debug logging for the name of the "key" file for that new cache slot; for example:
Then diff that key file against other key files in sibling directories to see what parameters changed. To determine which slot directories to compare, grep for the parse job IDs in "P" debug logging. You can get parse job IDs from annotation using Insight. Each key file contains an artificial == Ignored emake command-line options The following == Ignored environment variables By default, the following environment variables do not affect which cache slot is chosen. You can specify additional environment variables with EMAKE_PARSEFILE EMAKE_RLOGFILE ECLOUD_AGENT_NUM ECLOUD_BUILD_CLASS ECLOUD_BUILD_COUNTER ECLOUD_BUILD_ID ECLOUD_BUILD_TAG ECLOUD_BUILD_TYPE EMAKE_APP_VERSION EMAKE_BUILD_MODE ECLOUD_RECURSIVE_COMMAND_FILE WRAPPER_ARCH WRAPPER_BIN_DIR WRAPPER_BITS WRAPPER_CONF_DIR WRAPPER_FILE_SEPARATOR WRAPPER_HOSTNAME WRAPPER_INIT_DIR WRAPPER_JAVA_HOME WRAPPER_LANG WRAPPER_OS WRAPPER_PATH_SEPARATOR WRAPPER_PID WRAPPER_WORKING_DIR WRAPPER_SYSMEM_PP.P WRAPPER_NAME WRAPPER_DISPLAYNAME WRAPPER_DESCRIPTION WRAPPER_EVENT_JVM_ID WRAPPER_EVENT_JVM_PID WRAPPER_EVENT_NAME WRAPPER_EVENT_WRAPPER_PID == Ignored Android environment variables The following default set of environment variables do not affect which cache slot is chosen. This list is included as a default set only when you use |