KBEA-00129 - Using the parse avoidance feature

Article ID:360032825072
7 minute readKnowledge base

This KB article applies to ElectricAccelerator v7.0 and later.

Overview

ElectricAccelerator includes a parse avoidance feature, which can effectively eliminate makefile parse time in many builds. By caching and reusing parse result files, Accelerator can speed up both full builds and incremental builds.

Parse avoidance works when emulating GNU Make with clusters and/or local agents (such as with ElectricAccelerator Developer Edition).

Parse avoidance uses the concept of cache "slots." When parse avoidance is enabled, Electric Make (eMake) maintains a slot for each combination of non-ignored command line options, non-ignored environment variable assignments, and the current working directory. (Ignored command line options and environment variables are listed below.) A slot can be empty or can hold a previously-cached result. If the appropriate slot holds an up-to-date result, parsing is avoided.

If eMake detects file system changes that might have caused a different result (with the same command line options, environment variable assignments, and current working directory), a cached result becomes obsolete. Such file system changes include any file read by the parse job, which means all makefiles, all programs invoked by $(shell), the files they read, and so on.

Console Output

When a cached parse result is reused, eMake replays the file system modifications made during the parse job and any standard output or standard error that the original parse job produced. For example, if the makefile includes the following lines:

VARIABLE1 := $(shell echo Text > myfile)
VARIABLE2 := $(warning Warning: updated myfile)

then when using a cached parse result, eMake creates a file named myfile and prints Warning: updated myfile.

Before v7.1, eMake omitted any standard output or standard error that the original parse job produced.

Enabling Parse Avoidance

You must first run a "learning" build with the parse avoidance feature enabled. To enable parse avoidance, use the --emake-parse-avoidance=1 eMake option.

During the learning build, the option causes eMake to save the parse result to the (empty) cache. For subsequent builds, the option enables the reuse of cached parse results and saves a new result to the cache as appropriate. If you do not specify --emake-parse-avoidance=1, then the parse avoidance cache is not accessed at all.

The following actions are also recommended:

  • Turn on --emake-autodepend=1

  • Disable generated dependencies (either by modifying your makefiles or by using --emake-suppress-include as described below)

The following table describes options related to parse avoidance.

|== | eMake Option | Description

| --emake-assetdir=<`path>` | Use the specified directory for assets such as cached parse results. The default directory is .emake. This option also affects dependency optimization.

| --emake-parse-avoidance=<`0/1>` | Avoid parsing makefiles when prior parse results remain up to date. Cache new parse results when appropriate.

| --emake-parse-avoidance-ignore-env=<`var>` | Ignore the named environment variable when searching for applicable cached parse results. To ignore more than one variable, use this option multiple times.

| --emake-parse-avoidance-ignore-path=<`path>` | Ignore this file or directory when checking whether cached parse results are up to date. Append % for prefix matching. To ignore more than one path or prefix, use this option multiple times.

| --emake-suppress-include=<`pattern> ` | Skip matching makefile includes (such as generated dependencies). In general, you should not suppress makefile includes, unless they are generated dependency files and you have also enabled automatic dependencies as an alternative way of handling dependencies. Note: If the pattern does not have a directory separator, then the pattern is compared to the include’s filename component only. If the pattern has a directory separator, then the pattern is taken relative to the same working directory that applies to the include directive and compared to the included file’s entire path. |==

If a file is read during a parse, but it changes before eMake attempts to reuse that parse’s results, the cached parse result is normally considered to be obsolete. You can, however, override this decision temporarily by using --emake-parse-avoidance-ignore-path.

eMake permanently ignores the effect on a parse result of certain special files and directories that might have existed when it was created, in all cases using the names they would have had at that time:

  • Anything in the .emake directory or whatever alternative directory you specified using --emake-assetdir

  • Client-side eMake debug log (as specified by --emake-logfile)

  • Annotation file (as specified by --emake-annofile)

  • History file (as specified by --emake-historyfile)

Notes:

  • Your makefiles should avoid any $(shell) expansion that uses these four special files and directories in a meaningful way, because parse avoidance will ignore changes to them.

  • Enabling remote parse debugging (using the --emake-rdebug option) disables parse avoidance.

You can use a pragma to instruct parse avoidance to detect the dependence of a parse result upon path wildcard match results. This pragma makes the parse cache sensitive to the existence (or nonexistence) of specific files in directories that are subject to wildcard matching, as well as in new subdirectories thereof. (eMake does not detect matching files in just any new subdirectory, but only those that are subdirectories of directories read in the original job and their subdirectories, recursively.)

You can specify more than one glob pattern, as in the following pragma:

#pragma jobcache parse -readdir *.h -readdir *.c

For Android-specific information about parse avoidance and other Android best practices, see KB article KBEA-00130 at KBEA-00130 - Best Practices for Android Builds Using CloudBees Build Acceleration 10.0 and Older Versions.

Parse Avoidance Example

This example provides command line options for Android 4.1.1 (using the attached makefile):

--emake-parse-avoidance=1 --emake-autodepend=1 --emake-suppress-include=*.d --emake-suppress-include=*.P --emake-debug=P -f Makefile -f noautodep.mk
You might need additional options such as --emake-cm.

Deleting the Cache

To delete the cache, delete <`assetdir>/cache.. (The default asset directory is `.emake.) For example, enter rm -r .emake/cache.

Moving Your Workspace

If you want to move your workspace, make sure that the new eMake roots correspond to the old eMake roots. Also, because the asset directory defaults to .emake in the current working directory, you must either copy that directory to the new workspace or use --emake-assetdir= to specify an asset directory that you want the two workspaces to share. If you already use --emake-assetdir= to point to an asset directory within your old workspace and also want to move the asset directory, you must update its value to point to the new asset directory location.

eMake looks for eMake roots in the values of Make variables. When eMake replays a cached parse result in a new workspace, it replaces the old eMake roots in the values of those variables with the new eMake roots. This policy works well as long as no confusion exists between eMake roots and other text in those variable values. For example, if the value of a Make variable is -I/we will look into this, your old eMake root is /we, and your new eMake root is /wg, then the new value will be -I/wg will look into this. For best results, choose distinctive directory names for your workspaces.

Limitations

  • Windows is not supported

  • The parse avoidance feature does not detect the need to reparse in these cases:

    • Changes to input files and programs that are used during the parse (such as makefile includes) but that Accelerator does not virtualize (because they are not located under --emake-root=...).

    • Changes to non-filesystem, non-registry aspects of the environment (such as the current time of day)

  • If a parse job checks the existence or timestamp of a file without reading it, parse avoidance might not invalidate its cached result when the file is created, modified, or destroyed. This limitation is lifted if the file matches a glob pattern that you specify with “`#pragma jobcache parse ...and if the parse job performs a wildcard match on the containing directory. Using `--emake-autodepend=1 and --emake-suppress-include=<`pattern>` along with parse avoidance also helps to avoid this limitation. If a generated dependency file did not exist when a parse result was saved into the cache, eMake might reuse that cached parse result, even after that dependency file was created. Of course, you also benefit from the ElectricAccelerator eDepend feature’s performance and reliability gains.

Debugging

To log information about parse avoidance, enable parse avoidance debug logging. You do this by including a capital "P" in the argument of the --emake-debug=<`value>` option to set the parse avoidance debug log level. For more information about debug logging, see the "Troubleshooting" chapter of the ElectricAccelerator Electric Make Users Guide at https://docs.cloudbees.com/docs/cloudbees-build-acceleration/latest/emake-user-guide/troubleshooting.

Debugging Log Example

You can look in "P" debug logging for an explanation of why a cached result was obsolete:

WK01: 0.062173 Input changed: job:J08345218 slot:b6189634a793fee2fe1929fbf47cc4e4 path:/home/aeolus/t/Makefile thenSize:175 nowSize:176 ignore:0
WK01: 0.062228 Input changed: job:J08345218 slot:b6189634a793fee2fe1929fbf47cc4e4 path:/home/aeolus/t/fog.mk thenSize:0 nowSize:1 ignore:0
...
WK01: 0.062284 Cache slot is obsolete: job:J08345218 slot:b6189634a793fee2fe1929fbf47cc4e4

Notice that "Makefile" and "fog.mk" were both larger; either of those changes would have triggered a re-parse. The "ignore:0" comments indicate that --emake-parse-avoidance-ignore-path was not used to ignore the changes.

Using Key Files for Debugging

To discover why a new cache slot was used, look in the "P" debug logging for the name of the "key" file for that new cache slot; for example:

WK01: 0.250878 Saved context:
WK01: 11.853035 Saved slot definition: .emake/cache.11/i686_Linux/d9/0f/79/4fb975e8ff94dfa2569a303e62.new.2NPR8J/key
The ".new.2NPR8J" portion of the slot directory name should have already been removed automatically by eMake before you need to access the key file.

Then diff that key file against other key files in sibling directories to see which parameters changed. To determine which slot directories to compare, grep for the parse job IDs in "P" debug logging. You can get parse job IDs from annotation using ElectricInsight. Each key file contains an artificial "cd" command to express the working directory, all non-ignored environment variables, and the non-ignored command-line arguments.

Ignored command-line arguments and environment variables are omitted from key files. The key file is stored in a directory whose path name is derived from the MD5 digest of the file itself, which you can compute using the "md5sum" program. Also, the command line might have been normalized to one that is equivalent to the original one. eMake quotes special characters according to UNIX, Linux, or Cygwin "sh" shell rules.

In Accelerator versions before 7.1, the "key" file was named "environment".

Ignored Command Line Options and Environment Variables

The following command line options do not affect which cache slot is chosen:

|== | --emake-annodetail --emake-annofile --emake-annoupload --emake-assetdir --emake-big-file-size --emake-build-label --emake-class --emake-cluster-timeout --emake-cm --emake-debug --emake-hide-warning --emake-history --emake-history-force --emake-historyfile --emake-idle-time --emake-job-limit --emake-ledger | --emake-ledgerfile --emake-localagents --emake-logfile --emake-logfile-mode --emake-maxagents --emake-maxlocalagents --emake-mem-limit --emake-mergestreams (v7.0 only) --emake-parse-avoidance-ignore-path --emake-pedantic --emake-priority --emake-readdir-conflicts (v7.0.1+) --emake-resource --emake-showinfo --emake-tmpdir --emake-yield-localagents |==

The following environment variables do not affect which cache slot is chosen.

|== | BUILD_DISPLAY_NAME BUILD_ID BUILD_NUMBER BUILD_TAG CI_JENKINS_BUILD_ID CI_JENKINS_BUILD_NUMBER CI_JENKINS_BUILD_TAG CI_JENKINS_HUDSON_SERVER_COOKIE CI_JENKINS_JENKINS_SERVER_COOKIE COMMANDER_JOBID COMMANDER_JOBSTEPID COMMANDER_SESSIONID COMMANDER_WORKSPACE COMMANDER_WORKSPACE_UNIX COMMANDER_WORKSPACE_WINDRIVE COMMANDER_WORKSPACE_WINUNC COMMANDER_RESOURCENAME EMAKE_PARSEFILE EMAKE_RLOGFILE ECLOUD_AGENT_NUM ECLOUD_BUILD_CLASS ECLOUD_BUILD_COUNTER ECLOUD_BUILD_ID ECLOUD_BUILD_TAG ECLOUD_BUILD_TYPE EMAKE_APP_VERSION EMAKE_BUILD_MODE ECLOUD_RECURSIVE_COMMAND_FILE EMAKE_LEDGER_CONTEXT EMAKE_MAKE_ID TMP TEMP TMPDIR _ HOSTNAME OLDPWD SHLVL SSH_AGENT_PID SSH_AUTH_SOCK SSH_CLIENT | SSH_CONNECTION SSH_TTY GPG_AGENT_INFO COLUMNS LINES TERM TERMCAP COLORFGBG LS_COLORS DISPLAY DESKTOP_SESSION SHELL_SESSION_ID SESSION_MANAGER DBUS_SESSION_BUS_ADDRESS KDE_FULL_SESSION KDE_MULTIHEAD KDE_SESSION_UID KDE_SESSION_VERSION KONSOLE_DBUS_SERVICE KONSOLE_DBUS_SESSION WINDOWID WINDOWPATH XCURSOR_THEME XDG_SESSION_COOKIE XDM_MANAGED WRAPPER_ARCH WRAPPER_BIN_DIR WRAPPER_BITS WRAPPER_CONF_DIR WRAPPER_FILE_SEPARATOR WRAPPER_HOSTNAME WRAPPER_INIT_DIR WRAPPER_JAVA_HOME WRAPPER_LANG WRAPPER_OS WRAPPER_PATH_SEPARATOR WRAPPER_PID WRAPPER_WORKING_DIR WRAPPER_SYSMEM_PP.P |==

You can specify additional environment variables by using --emake-parse-avoidance-ignore-env. You must use --emake-parse-avoidance-ignore-env once for each variable to ignore.

Applies to

  • Product versions: 7.0 and later

  • Platforms supported: Linux