Managing Temporary Files

3 minute read

During the build process, temporary files are created, then deleted automatically when a build completes. To maintain optimum efficiency, it is important to identify where these files are created in relation to the final output files.

Configuring the eMake Temporary Directory

eMake runs commands and collects output in parallel, but the results are written in order, serially. This feature ensures a build behaves exactly as if it were run locally. See Transactional Command Output.

The eMake temporary directory is used to store output files from commands (for example, object files) that finished running but are waiting their turn to be relocated to a final destination on disk. By default, eMake creates a temporary directory in the current working directory. The directory name will have the form:

ecloud_tmp_<pid>_<n> where pid is the eMake process identifier. where n is a counter assigned by eMake to differentiate multiple temporary directories created during a single build if you specify multiple directories.

eMake removes the temporary directory on exit (including the Ctrl-C user interrupt). If eMake is terminated unexpectedly (for example, by the operating system), the temporary directory might persist and need to be removed manually.

Important temporary directory requirements include:

  • Must be writable by the eMake process

  • Must not be on an NFS share

  • Must have enough space to contain the build output

  • Ideally, the temporary directory should have enough space to hold the complete build output; in practice however, it might need only enough space for output from the largest commands.

  • The exact temporary directory space requirement varies greatly with the build size, the number of Agents used, and build system speed.

You can use the --emake-tmpdir command-line option or the EMAKE_TMPDIR environment variable to change the temporary directory default location:

% emake --emake-tmpdir=/var/tmp ...

Files are relocated from the temporary directory to their final location. Keeping the temporary directory on the same file system as the EMAKE_ROOT helps performance because files can simply be renamed in place, instead of copied.

eMake and agent machines reset the values for TEMP, TMP, and TMPDIR. This is necessary to avoid possible conflicts with multiple build agents/jobs running on the same machine.

If you have more than one EMAKE_ROOT that spans multiple file systems, you can specify more than one temporary directory to ensure eMake can always rename files in place, instead of copying them.

For example, if your EMAKE_ROOT contains three directories:

% setenv EMAKE_ROOT /home/alice:/local/output:/local/libs

that reside on two different file systems

% df /home/alice /localFilesystem     1K-blocks  Used   Available Use% Mounted onfiler:/vol/    76376484 47234436 25262352   66%  /homevol0/space /dev/hdb1      76896316 51957460 21032656    2%  /local

to ensure better performance, specify two temporary directories—write permission at both locations is required:

% emake --emake-tmpdir=/home/alice:/local ...

When specifying multiple temporary directories, note the following:

  • If a temporary directory was specified for a particular file system, eMake automatically uses that directory for any files destined to reside on that file system.

  • If a temporary directory was not specified for a particular file system, eMake uses the first directory specified for files destined to reside on that file system.

Deleting Temporary Files

During a build, eMake creates a temporary directory inside the directory specified by the EMAKE_TMPDIR environment variable or in the directory specified by the --emake-tmpdir command-line option (or the current working directory if no eMake temporary directory is specified). If a specified directory does not exist, eMake creates it. All temporary directories created by eMake are automatically deleted when a build completes.

For example, if you set your temporary directory to /foo/bar/baz and only /foo/bar exists, eMake creates /foo/bar/baz and /foo/bar/baz/ecloud_tmp_<pid>_<n>, and then deletes both directories and all their contents when it exits. If /foo/bar/baz exists at the start of the build, only /foo/bar/baz/ecloud_tmp_<pid>_<n> is created and deleted.

However, if the build is aborted in-progress, eMake will not have the opportunity to remove the temporary directory.

Subsequent eMake invocations automatically delete temporary directories if they are more than 24 hours old. You can reclaim disk space more quickly by deleting temporary directories and their contents by hand. However, do not delete the temporary directory while a build is in-progress —this action causes the build to fail.

Temporary directory names are in this form:

ecloud_tmp_<pid>_<n> where pid is the eMake process identifier. where n is a counter assigned by eMake to differentiate multiple temporary directories created during a single build if you specify multiple directories.