CloudBees is pleased to announce the availability of the CloudBees Build Acceleration 12.0 long-term support release.

CloudBees Build Acceleration is a software build accelerator that dramatically reduces build times by distributing the build over a large cluster of inexpensive servers. CloudBees Build Acceleration uses a patented dependency-management system to identify and fix problems in real time that break traditional parallel builds. CloudBees Build Acceleration plugs seamlessly into existing software development environments and includes web-based management and reporting tools.

CloudBees Build Acceleration includes the following components:

  • Electric Make (“eMake”)

  • Electric File System (EFS)

  • CloudBees Build Acceleration agents (“Agents”)

  • Cluster Manager

  • Electrify

Security fixes

Updates to third-party packages (EC-14004, EC-14049)

In this release the Cluster Manager is updated to use Apache 2.4.46, PHP 7.4.12, and OpenSSL 1.1.1i.

New features

Redesigned and expanded Build Details

The Build Details page in the Cluster Manager has been redesigned and expanded to include performance analysis and visualizations derived from CloudBees Build Acceleration Insight. This enhancement enables users and administrators to easily access the most critical performance data for builds from any system that can access the Cluster Manager web interface, even if the annotation file for the build is not accessible. The Build Details page now includes several new sub-tabs:

  • Timeline: a visualization of the execution of jobs in the build.

  • Composition: analysis of the type of work performed in the build, such as compilation or packaging, and the portion of the total build workload represented by each.

  • Jobcache: analysis of the use and performance of jobcache in the build, including the cache hit rate and estimated time savings due to jobcache.

  • Performance: general performance information, including the number of agents in use over the lifetime of the build and a prioritized summary of the build critical path so users can quickly identify those jobs that have the greatest impact on the build duration.

  • Diagnostics: warnings and errors emitted by the build, determined by intelligent analysis of the build output log.

  • Recommendations: actionable advice for improving build performance based on analysis of the build and identification of several known performance anti-patterns, such as missing or incomplete history, jobcache misses and more. Recommendations are prioritized according to the estimated impact on performance.

Other Cluster Manager enhancements

In addition to the updates described above, the Cluster Manager includes a number of other improvements:

  • For running builds, the Builds and Build Details pages in the Cluster Manager have been enhanced to show the build percentage complete.

  • When editing Resources, the Hosts text box is now a pull down with known agent hosts, to simplify the task of defining resources.

  • A Description column has been added to the Resources page.

  • Users can specify a different number of maximum items on the Builds and Agents pages.

Android support improvements

This release extends Android support to include builds based on AOSP 11. In addition, this release features several performance improvements for both full and incremental Android builds. For best results, users should regenerate their history files by running a full build with the --emake-history=create option; subsequent builds can be run without that option or with the value set to merge. The changes made include:

  • The javac jobcache type will now apply to jobs that use kotlinc and known aliases for the Kotlin compiler. (EC-13917)

  • A custom hash was introduced for Unix archive files which ignores member timestamps. This allows for cache hits when the objects in the archive are unchanged except for their modification time. (EC-14020)

  • A custom hash was introduced for C/C++ source files and headers which ignores comments and blank lines. This allows for cache hits when the code in source files is unchanged even if the comments are different. This is especially beneficial for generated source files, which frequently embed the time of generation in a comment. (EC-11533)

  • The jobcache feature was modified to omit explicit timestamp modification from the cached operations, so that outputs extracted from the cache will be assigned a current time stamp when they are created. This avoids unexpected rebuilds in incremental builds after full builds that used the jobcache. (EC-13699)

  • The ninja parser no longer creates jobs for prerequisites that are mentioned only in .ninja_deps. This avoids the cost of job processing for source files. (EC-13689)

  • emake no longer stat’s order-only prerequisites during up-to-date checks. (EC-13923)

  • The Android integration pragmafiles (enabled when you use the --emake-android-version option) have been retuned for Android 10 and Android 11 to avoid spurious rebuilds of up-to-date outputs during incremental builds and to reduce startup time. (EC-13916, EC-13915)

Bitbake and Yocto improvements

This release also includes enhancements to Bitbake and Yocto support to expand compatibility and improve preformance. The changes include:

  • Yocto 3.2 ("gatesgarth") is now supported.

  • Parse avoidance is enabled by default and emake is used to build more packages.

  • The integration now supports building packages that use ninja instead of make, via the ninja emulation mode in emake. The default configuration is updated to take advantage of this change, resulting in improved performance for Bitbake/Yocto users. (EC-13879)

  • The bb2anno utility has been enhanced to aggregate the data in <metrics> sections of the individual annotation files that are incorporated into the combined annotation file, to improve "whole build" performance analysis. (EC-13956)

Security configuration

This release adds support for configuring agent security settings during installation and after installation with ecconfig. Security settings configured using these methods will be preserved during future upgrades. Prior to this change, configuring agent security settings required modification to the agent startup scripts, and those settings were not retained during upgrades (EC-13407).

Automatic output directory creation

This release introduces a new command-line option for emake, --emake-create-output-dirs. When enabled, this will cause emake to automatically and efficiently create output directories for targets when running in GMake emulation modes. In addition, the details of the implementation are such that this will avoid provoking conflicts even in builds that have no history file or have an incomplete history file. This feature is a superior alternative to mechanisms such as mkdir -p $(dirname $@), directory creation sentry files, and others that have commonly been employed with make-based build systems to solve the problem of creating output directories for build targets (EC-13680).

Build signatures and totality

This release adds "build signatures", a hash computed from the serial ordered list of output targets in the build. This value can be used to identify builds that are substantively "the same" to facilitate comparisons and identification of trends across sets of like builds. The hash uses emake-root-relative target names, so the signature will be the same even if the absolute location of the build has changed (for example, as in continuous integration builds, or between developers working on the same project from their own copies of the source tree). The build signature for a build can be found in the annotation properties.

This release also adds a new "totality" metric, which is the percentage of jobs that were not up-to-date in the build. For a full, from-scratch build, this metric should be 100 in most builds. For a "no touch" build, this metric should be close to zero. For incremental builds, the value will be somewhere between those values, depending on the portion of the output that was rebuilt.

Insight updates

This release includes an updated release of CloudBees Build Acceleration Insight with several enhancements to support the new Build Details page in the Cluster Manager:

  • A new Recommendations report which analyzes build performance and provides recommendations of ways to improve performance. (EI-846)

  • A new JobCache Summary report which provides an overview of jobcache use in the build including an estimate of the elapsed time saved and the types of jobcache used. (EI-845)

  • A new Diagnostics report identifies error and warning messages in the build output log. (EI-843)

  • The Agent Utilization report now shows agent utilization over the lifetime of the build, instead of a histogram-like summary of utilization. This is both more intuitive and more useful since it allows the user to quickly see when the number of agents in use was low during the build.

In addition this release includes a variant of CloudBees Build Acceleration Insight specifically for command-line use on Linux, called einsight-cmd. This variant has no dependencies on GUI libraries, making it ideal for use in "headless" automation contexts.

Other improvements

This release includes an enhancement to the way symbolic links are handled, such that timestamps on symlinks created during the build are in the correct relative order when compared to timestamps on regular files created during the build. This avoids spurious rebuilds of up-to-date outputs during incremental builds. (EC-13444)

This release adds a new <configuration> section to annotation files, which describes the actual values used for every user-configurable option in emake. This eliminates ambiguity about the settings used in each build by providing a clear record that can be referenced when debugging or during performance analysis.

New platform support

Support is added for the following platforms:

  • CentOS 8.3

  • CentOS 8.2

  • CentOS 7.9

  • Debian 10.7

  • Debian 10.6

  • Debian 10.5

  • Debian 9.13

  • Red Hat Enterprise Linux 8.3

  • Red Hat Enterprise Linux 7.9

  • SUSE Linux Enterprise Server 15.2

  • Ubuntu 20.10

  • Windows Server 2019

In addition, the following platforms are now deprecated: these platforms are supported in 12.0, but they will not be supported in future releases:

  • Red Hat Enterprise Linux 5.x

  • Red Hat Enterprise Linux 6.0 through 6.9

  • ClearCase 7 and 8

  • 32-bit ClearCase 9

Furthermore, after the 12.0 release, electrify will not support monitoring 32-bit applications.

Finally, after the 12.0 release, the minimum browser requirement for the Cluster Manager web interface will be updated to the following versions:

  • Chrome 57

  • Edge 16

  • Firefox 52

  • Safari 11

No database support changes.

No browser support changes.

Resolved issues

emake assertion over willConflict flag (EC-13932)

With this fix, instead of failing with an assertion, emake will treat a job as in conflict if previous job activity indicated that the job should be in conflict, even if the file usage at the end of the job would not otherwise indicate a conflict.

electrify does not escape regex characters in --electrify-remote option (EC-13097)

With this fix, electrify treats the values specified in --electrify-remote as literal characters as expected, rather than as regex characters. This means that the user can specify --electrify-remote=g`, for example, and it will match the literal string `g rather than "g repeated one or more times". Use --electrify-allow-regex to specify process names using regex syntax.

Error when upgrading auto-scaling cloudburst agents (EC-13880)

With this fix, the installer correctly handles upgrades of agent hosts that are configured to automatically select the number of agents at startup, as is typical for cloudburst agent machine images.

Jobcached jobs do not update autodep data (EC-13924)

With this fix, emake correctly updates autodep information based on usage from jobs pulled from the jobcache.

Not possible to clear autodep data for a job (EC-13925)

Prior to this change there was no way to completely expunge the autodep information for a single job, even if the build was changed such that autodep no longer applied to the job. With this fix, emake will clear the autodep entry for the job if it is no longer applicable.

Build hangs waiting for cloudburst agents (EC-13805)

A race condition sometimes caused the Cluster Manager to fail to remove the database entries for agents corresponding to cloudburst instances that had been terminated. With this fix, the race condition is eliminated so the database is updated reliably.

Cloudburst agent instances are not always terminated automatically (EC-13072)

If the Cluster Manager was restarted while cloudburst agents were active, the Cluster Manager would lose track of the agents after the restart. The Cluster Manager would subsequently fail to terminate those agent instances, and would fail to reuse the agents for new builds. With this fix, the Cluster Manager correctly resumes management of cloudburst agents after restarting, ensuring that the agents are reused when appropriate and that the agents are terminated as expected.

bb2anno tool generates corrupted annotation (EC-13954)

If the annotation file for a task did not include at least waiting annotation detail and the annotation included recursive make invocations, bb2anno would erroneously truncate the task annotation data at the end of the first recursive make when incorporating it into the combined output annotation. With this fix, bb2anno instead detects task annotation files that do not have waiting annotation detail, prints a warning, and excludes those annotation files from the combined annotation.

Deleting builds from the Cluster Manager deletes too much from build_logs directory (EC-13982)

A defect in the Cluster Manager where deleting the logs for builds that were deleted via the Cluster Manager web interface or with cmtool resulted in the deletion of build logs for unrelated builds. With this fix, only the logs for the deleted builds are deleted.

Cluster Manager starts more agent containers than Kubernetes can sustain (EC-13660)

When using Kubernetes cloudbursting, the Cluster Manager failed to specify a CPU count when making agent container requests. This could result in the Kubernetes cluster trying to run more containers on a host than that host could reasonably support. With this fix, the Cluster Manager specifies a CPU count when starting new agent containers, enabling Kubernetes to correctly limit total deployment and balancing loads across hosts in the Kubernetes cluster.

Behavior changes

The jobcache version has been updated to version 34, as some of the enhancements in this release are incompatible with the previously recorded data in the cache. Builds using the new version will automatically repopulate the cache again from scratch. Users may wish to manually delete the old cache data to reclaim disk space.

With this release, emake no longer records explicit timestamp modifications, such as those arising from commands like touch -t …​ in operations saved to jobcache slots. This is consistent with other build caching tools, and eliminates a class of problems relating to replaying old file timestamps in new builds (EC-13699).

As part of CloudBees' ongoing commitment to inclusivity, the bemake configuration files used to control which bitbake packages should be built using emake and which should be built using gmake have been renamed to bemake_package_accelerated and bemake_package_exceptions, respectively. For backwards compatibility the legacy names bemake_package_whitelist and bemake_package_blacklist will be used if files matching the new names cannot be found, but the legacy names are considered deprecated and anybody using CloudBees Build Acceleration to build using bitbake is encouraged to rename these files in their setup, if present. (EC-13892)

Installation and upgrade notes

Cloud bursting

Because the agent to Cluster Manager communication uses TLS by default, the agent image used for AWS EC2, GCP or Azure cloud bursting must also be upgraded to the 11.3 release or later in order for cloud bursting to work.

New CloudBees Build Acceleration release strategy

Starting in August 2019, the release strategy for CloudBees Build Acceleration is updated to add a “preview” release in addition to the standard long-term support (LTS), maintenance (patch), and hotfix releases.

The release numbering for the preview releases uses a new <year>.<month>.00 numbering scheme. For example, 2020.08.00.

CloudBees recommends that you upgrade to the preview release and test these features in a controlled environment before rolling them out to production.

For details about the new release strategy, see the CloudBees Build Acceleration Release Strategy Update web page.

Version enforcement in Cluster Manager licensing

As of version 11.1, perpetual license files include a Version field and are therefore no longer transferable to succeeding product versions. At run time, to be considered valid by the Cluster Manager, the license file must have a Version that equals or exceeds the version of the Cluster Manager itself. Version checks consider only the major.minor version (for example, a license for 11.1 is valid for 11.1.0, 11.1.1, 11.1.2, and so on).

eMake authenticates with the Cluster Manager via the eMake/Cluster Manager protocol version to ensure that the eMake and Cluster Manager licenses match. If the license has no Version field, it will be considered invalid by the 11.1 Cluster Manager.

You can upgrade just the Cluster Manager, but starting with version 11.1, if you upgrade eMake or agents, you must also upgrade the Cluster Manager. When you upgrade the Cluster Manager, you must also acquire a new license if you are currently using a perpetual license.

You can upgrade to new patch releases (but not new feature releases) without acquiring a new license. For example, if your current license specifies 11.1, then you can use a Cluster Manager with version 11.1, 11.1.1, 11.1.2, and so on. Also, for example, if your license specifies 12.0, then you can use a Cluster Manager with version 12.0, 12.0.1, 12.0.2, and so on, as well as 11.x.

The Administration > Licenses page in the Cluster Manager web UI has a Version column, which provides the product version of each license in the list. This information is blank for a license with no explicit version (such as an evaluation license).

Make sure that you import a compliant license into the Cluster Manager before upgrading. For assistance, see CloudBees Support. For details about importing a license, see Logging In and Enabling Licensing - Importing Your License. (EC-13170 and EC-13154)

Hardware requirements

The recommended total amount of RAM for an agent host is 2 GB per agent plus the amount of RAM normally needed to execute your build. For example, if you are running four agents, and your build normally needs 16 GB, you will need ((2 * 4) + 16) = 24 GB.

Backing up before you upgrade

The upgrade process does not preserve the existing files. Back up the /opt/ecloud/<arch>/cloud directory for Linux or the C:\ECloud\<arch> folder for Windows to a safe location.

For additional security, back up the database by following the recommended procedure from your database vendor.

Installing JDBC drivers for MySQL or Oracle databases

CloudBees no longer distributes the JDBC drivers for MySQL or Oracle databases. To use one of these databases, you must download its driver directly from the Oracle website, then copy it to the appropriate directory on the Cluster Manager server, and then restart the Cluster Manager service. For details, see Installing JDBC Drivers for MySQL or Oracle Databases.

If you relocate eMake

If you copy the emake executable to a new location, you must also copy the execserver executable to that location. By default, the path to the execserver executable is /opt/ecloud/i686_Linux/32/bin/execserver (for 32-bit eMake) or /opt/ecloud/i686_Linux/64/bin/execserver (for 64-bit eMake).

Regenerating history files after an upgrade

The identifier that is used to find certain types of jobs in the eMake history file changed in version 8.0. After an upgrade from version 7.2.2 or older versions to version 8.0 or newer versions, users should regenerate their history files by running their first build with the --emake-history=create option to avoid unnecessary serializations. This build might have more conflicts than normal (but subsequent builds should return to normal).

Concurrent build licensing

As of version 9.1, for new CloudBees Build Acceleration subscription licenses, the number of builds that you can run concurrently is license-limited. The noLicenseWaitTime performance metric indicates the amount of time that a build spent waiting for a concurrent build license because the number of concurrent builds reached the license limit. Also, as of version 9.1, JobCache is not separately licensed and is now included with the concurrent build license.

Customers using pre-9.1 CloudBees Build Acceleration licenses may continue to use those licenses, including the licenses for the JobCache add-on.

For details about licensing for concurrent builds, see the Logging In and Enabling Licensing. (EC-12095)

Known issues

Linux kernel issue that affects CloudBees Build Acceleration performance

Affected kernel versions:

  • RHEL kernel versions later than 2.6.18-194.32 and earlier than 2.6.32-131

  • Ubuntu Linux kernel versions 2.6.31, 2.6.32, 2.6.33, and 2.6.34

Symptoms:

Affected systems might encounter reduced performance on both ext3 and ext4 file systems. Symptoms might include

  • hung_task_timeout_secs messages in system dmesg logs

  • Widely variable agent availability (entering and exiting agent “penalty” status frequently)

  • Contention over the ecagent.state file

  • Slower builds (with unexplained variances)

To help determine if this issue exists, run the dmesg | grep hung_task_timeout command. hung_task_timeout errors show that this issue is present. Contact your kernel provider for another version of the precompiled kernel.

Fixes for systems running RHEL 5.6, 5.7, 5.8, and 6.0

You should consider upgrading to 2.6.32-131 (RHEL 6.1) or downgrading to 2.6.18-194.32 (RHEL 5.5).

Other known issues
  • Android M is not compatible with CloudBees Build Acceleration 11.1. You must use Android M with CloudBees Build Acceleration 10.0.

  • If you kill a build manually, and the agents running on an Amazon EC2 instance fail to connect, the instance will continue to run. You must kill the instance manually.

  • If the Cluster Manager goes down while agents are running on an Amazon EC2 instance, they will still connect but will not be associated with a resource. You must kill the corresponding instance manually.

  • The cmtool importData command does not import license properties (such as maxAgents ). To work around this issue, re-import the license after using importData. (EC-12371)

  • You cannot control breakpoints from the Cluster Manager. (EC-12322)

  • If Apache fails to start properly after a new Cluster Manager installation, reboot the system.