CloudBees Build Acceleration Terms and Concepts

8 minute read

Welcome to CloudBees Build Acceleration®

CloudBees Build Acceleration from CloudBees, Inc. is a software build accelerator that dramatically reduces build times by distributing builds over numerous inexpensive servers on a LAN or a public cloud such as Amazon EC2 or Google Cloud Platform. It uses CloudBees’ proven technology to speed up builds and manage build workload. The purpose of CloudBees Build Acceleration is to enable more-frequent build-test cycles and shorten the time to market.

CloudBees Build Acceleration provides optional JobCache functionality, which enables the reuse of build outputs from one build to the next to further accelerate builds. For details, see Job Caching.

CloudBees Build Acceleration uses a patented dependency management system to identify problems that break parallel builds and fix them in real time. Build speeds vary as source files grow and change—and CloudBees Build Acceleration continuously studies actual build and test dependencies and uses that information to reorder steps dynamically and optimally for subsequent builds.

CloudBees Build Acceleration plugs seamlessly into existing software development environments and includes web-based management and reporting tools. CloudBees Build Acceleration includes the following main components:

This document includes some basic terms and concepts as well as explanations of the relationships among the parts of CloudBees Build Acceleration. It also includes more information about the CloudBees Build Acceleration Cluster Manager Web UI—a web-based dashboard for managing the pool of servers and viewing numerous types of build-related activity.

Integration with Your Existing Environment

CloudBees Build Acceleration works well with a variety of build environments and tools such as:

  • Popular Windows and Linux operating systems as well as IDEs and test tools. CloudBees Build Acceleration plugs into your existing tools and scripts to allow push-button preflight and production builds and tests without modifying your development and test environment.

    For Visual Studio users, the CloudBees Build Acceleration installer adds the ACCELERATOR menu to the Visual Studio menu bar to let you run builds directly from within Visual Studio at the click of a button. You do not need to convert your project or solution files.

  • Existing GNU Make (gmake), gcc, Microsoft NMAKE, Cygwin, and other build infrastructures. CloudBees Build Acceleration relies on eMake—a high-performance replacement for gmake. You invoke eMake interactively or through build scripts.

  • The Ninja build system. CloudBees Build Acceleration can emulate Ninja in addition to standard CloudBees Build Acceleration make-based builds such as gmake and NMAKE.

  • Popular static code analysis tools such as those from Klocwork, Parasoft, and Coverity as well as linting tasks or unit tests.

For best performance and results, your development and test environments should be identical across all participating servers.

Cluster

A cluster is a set of cores in a pool of servers that participate in distributed builds. For details about clusters, see Understanding Component Interactions.

Cluster Manager

The Cluster Manager is software that is installed on a server to manage a cluster. The Cluster Manager monitors system activity and manages eMake. The Cluster Manager also hosts the Cluster Manager Web UI. For more information about the Cluster Manager, see Understanding Component Interactions.

Build Classes

Build classes are classifications that you can use to group builds by version or release, product type, development stage, platform, or other criteria to organize your build management process. You use the Cluster Manager Web UI to create build classes. For details about build classes, see Creating Build Classes and xref:[Using Build Classes].

Resources

A resource lets you partition your Cluster to service different types of builds or groups of users based on operating system type, machine capacity, or business unit. You use the Cluster Manager Web UI to manage resources. For details about resources, see Resources.

Fault Tolerance

If a participating cluster machine is off the network (such as when it is shut down), CloudBees Build Acceleration reassigns incomplete jobs automatically to the next available participating machine to complete the build.

Electric File System

Electric File System (EFS) is a special file system driver that monitors every file access on participating machines and provides eMake with complete usage information. EFS collects dependency information, which lets eMake detect and correct out-of-order build steps. For details about the EFS, see Understanding Component Interactions.

Agents and Agent Hosts

Each EFS driver instance is paired with an agent, which is the user-level component running on the machines on which each agent is installed (“agent hosts”). An agent is the intermediary between eMake and EFS. One agent per core is recommended.

As the user-level component running on the hosts, an agent is inseparable from EFS—an agent is an intermediary between eMake and EFS. Agents run commands on behalf of eMake. Depending on your system configuration, you might have one or slightly more EFS-agent pairs installed per core. You use the Cluster Manager Web UI to manage agents and agent hosts. For more information about the role of agents in CloudBees Build Acceleration, see Understanding Component Interactions.

eMake

eMake is a new Make version and is the main build application. eMake reads makefiles in several different formats, including gmake and NMAKE. eMake distributes commands to the Cluster for remote execution and also services file requests. For details about eMake, see Understanding Component Interactions.

Makefiles

eMake executes makefiles just like gmake. CloudBees Build Acceleration uses CloudBees’ conflict detection technology to build faster than the gmake parallel make process without the broken builds that commonly occur with gmake.

CloudBees Build Acceleration detects the actual dependencies for complex builds. It eliminates broken or incorrect builds by reordering steps dynamically to fix dependency problems that occur when makefiles are generated by other processes.

During build execution, CloudBees Build Acceleration virtualizes your build file system dynamically so that agents running on different hosts can read and write build files without requiring the setup and maintenance of a separate network file system such as NFS or CIFS. eMake is a lightweight file server to the agents that are actually running commands on its behalf. Advanced caching, file compression, and peer-to-peer sharing between agents reduce network overhead to the absolute minimum.

For more information about using makefiles with CloudBees Build Acceleration, see the Electric Make User Guide.

eMake Root

The "eMake root" identifies the parts of the eMake machine’s file system that will be accessible to CloudBees Build Acceleration. The --emake-root option specifies the location of the Electric File System root directory (or directories). All files under the eMake root directory (or directories) are mirrored automatically on each agent.

You must specify the correct root directory (or directories), or the build might fail, because eMake cannot find the necessary files. (If the --emake-root option is not identified explicitly, then eMake will use only the current directory.) The eMake root must include any files written by the build, should include any source files read by the build, and may include tools used by the build. For best results and performance, be sure to include:

  • All files written by the build

  • All source files

  • The location for build output files (such as object files and linker output)

  • Other files read by the build (such as third-party tools and system headers)

To specify more than one directory or subdirectory, separate them by using standard PATH variable syntax (a colon for UNIX or a semicolon for Windows). For example, on UNIX:

--emake-root=/src/foo:/src/baz

This example streamlines the root path by excluding other /src subdirectories not needed by the build.

Any files used by the build, but not included under an eMake root directory, must be preloaded onto all hosts and identical to corresponding files on the system running eMake. If these files are not identical, eMake could find and use the wrong files. This approach is appropriate for system compilers, libraries, header files, and other files that seldom change and are used for all builds.

An improperly set eMake root is the most common cause of unexplained build errors and broken builds. For more information about the eMake root, see Setting the eMake Root Directory.

Licensing

CloudBees Build Acceleration is licensed based on the number of agents in use, the entitled version, and, for most licenses, the number of builds that are running concurrently. You use the Cluster Manager Web UI to import a license to enable licensing. For more information about licensing, see Logging In and Enabling Licensing.

Cluster Manager Web UI

The Cluster Manager web UI is installed as part of the Cluster Manager. It is accessible at https://<server_hostname_or_IP>/.

You use the Cluster Manager web UI to see the list of builds and other build information, reports about agents, and other statistics. The following sections describe some of the key functionality of the web UI.

Home Tab

The Home tab provides information such as links to documentation and other assistance, instruction for novice users, and charts for agent usage and build durations during the past 24 hours.

cm home tab

For details about this tab, see The Home Page.

List of Builds and Other Build Information

The Builds tab displays a list of the running and completed builds and key information about each build such as the start time and duration, the hardware platform, and whether completed builds succeeded or failed.

cm builds tab

For details about this tab, see Builds.

The Build Classes subtab lets you view and edit the classifications that you use to group builds by version or release, product type, development stage, platform, or other criteria to organize your build management process. For details about this subtab, see the Build Classes.

Details About Agents

Click the Agents tab to see details about each Agents and Agent Hosts in the Cluster such as the agent host name and hardware platform, agent software version, and agent status (such as whether it is running, has a problem, or has failed).

cm agents tab

For details about this tab, see Agents.

The Resources subtab lets you perform cluster resource management tasks such as creating, editing, or deleting Resources. For details, see Resources.

The Agent Policies subtab lets you manage a cluster by configuring agent allocation and the clock skew (out-of-sync) allowed between eMake clients and agents in the cluster. For details, see the Agent Policies.

Reports and Build and Agent Statistics

Click the Reports tab to see reports using many types of analytics about your build cluster and several types of data visualization such as scatter plots and pie charts.

cm reports tab

For example, the Build Duration report is a scatter plot that shows the durations of builds over a specified time period:

cm reports tab build dur

For details about the Reports tab, see Reports. For details about the Build Duration report, see Build Duration Report.

You can click a build to navigate to the Build Details page for that build:

cm reports tab build det

For details about this page, see Build Details.

The Server Load subtab provides metrics such as the number of running builds and agent statistics such as agent shortages caused by high build demand or license limits. For details about this subtab, see Server Load.

The Realtime Server Metrics subtab provides metrics such as JVM free memory and the number of concurrent builds. This tab provides low-level performance metrics and is meant primarily for use by CloudBees Technical Support.

Cluster Manager Messages

Click the Messages tab to see messages from the Cluster Manager, Agents and Agent Hosts, and eMake. This tab provides message details such as the severity, time and date, message text, and the build or agent that reported the message.

cm messages tab

For details about this tab, see Messages.

The Message Policies subtab lets you configure the minimum severity level for email notifications and whether to be notified when new messages are generated. For details about this subtab, see Message Policies.

Agent Cloud Bursting

Click the Cloud tab to see status for each Azure Services Platform, Amazon EC2, Google Cloud Platform, or Kubernetes instance that is active (that is, running a build). This tab provides details such as the instance ID, status (such as pending, running, or done), host name (internal DNS name), platform, and date and time the instance was launched.

For details about this tab, see Cloud Burst Status.

Cluster Manager Administration

Click the Administration tab to view and edit details about the current logged-in user (such as the full name, email address, and assigned groups).

cm admin tab

For details about this tab, see Administration.

The Permissions subtab lets you view and edit a variety of permissions for cluster users, groups, and the Cluster Manager administrator. You use this page to set or modify the permissions for using other tabs in the Cluster Manager web UI and for Cluster Manager server access, user impersonation, and eMake invocation.

cm admin perms subtab

For details about this subtab, see Permissions.

The Server Settings subtab displays Cluster Manager server preferences for message logging, notifications, transmission of anonymous usage statistics to CloudBees, and management of optional cloud providers such as Amazon EC2 or Kubernetes.

cm admin svr stngs subtab

For details about this subtab, see Server Settings.

The Licenses subtab displays all license information known to the Cluster Manager and shows the usage to which you are entitled. The Cluster Manager reads this information from the current license file. This page typically displays a single license.

cm admin lic subtab

For details about this subtab, see Licenses.

The Users, Groups, and Database Configuration subtabs let you add, delete, or configure users and local user groups and configure the database connection (name, host name and port, login credentials, and database type such as MariaDB or Oracle). For details about these subtabs, see Users page, Groups page, and Database Configuration.