Understanding Component Interactions

2 minute read

To a user, CloudBees Accelerator might appear identical to other Make versions—reading makefiles in several different formats and producing identical results. Using a cluster for builds is transparent to the Accelerator user.

Important differences in Accelerator build processing versus other distributed systems:

  • Components work together to achieve faster, more efficient builds. Instead of running a sequential build on a single processor, Accelerator executes build steps in parallel on a cluster of hosts.

  • Components work together to achieve faster, more efficient builds. Instead of running a sequential build on a single processor, CloudBees Accelerator Developer Edition executes build steps in parallel using multiple local agents.

  • For fault tolerance, job results are isolated until the job completes. If an Agent fails during a job, Accelerator discards any partial results it might have produced and reruns the job on a different Agent.

  • Missing dependencies discovered at runtime are collected in a history file that updates each time a build is invoked. Accelerator uses this collected data to improve performance of subsequent builds.

eMake and EFS

High concurrency levels in Accelerator are enabled by the Electric File System (EFS). When a job such as a compilation runs on a host, it accesses files such as source files and headers through EFS. EFS records detailed file access data for the build and returns that data to eMake.

eMake acts as a file server for Agents, reading the correct file version from file systems on its machine and passing that information back to the Agents. Agents retain different file version information and do not rely on eMake’s file sequencing ability to provide the correct version for a job. The Agent receives file data, downloads it into the kernel, notifying EFS, which then completes the original request. At the end of a job, Electric Agent returns any file modifications to eMake so it can apply changes to its local file systems.

eMake and Cluster Manager

When eMake is invoked on the build machine, it communicates with Cluster Manager to acquire a set of Agents it can use for the build. When eMake finishes, it sends Cluster Manager the build results, and tells Cluster Manager that Agents are free now to work on other builds. If more than one build is invoked, Cluster Manager allocates agents using a priority-based algorithm. Builds with the same priority share Agents evenly, while higher priority builds are allocated more Agents than lower priority builds. By default, agents running on the same host machine are allocated to the same build. In real time, Cluster Manager dynamically adjusts the number of agents assigned to running builds as each build’s needs change, which allows Accelerator to make the best use of cluster resources.