CloudBees is pleased to announce the availability of the CloudBees Build Acceleration 12.1 long-term support release.
CloudBees Build Acceleration is a software build accelerator that dramatically reduces build times by distributing the build over a large cluster of inexpensive servers. CloudBees Build Acceleration uses a patented dependency-management system to identify and fix problems in real time that break traditional parallel builds. CloudBees Build Acceleration plugs seamlessly into existing software development environments and includes web-based management and reporting tools.
CloudBees Build Acceleration includes the following components:
Electric Make (“eMake”)
Electric File System (EFS)
CloudBees Build Acceleration agents (“Agents”)
- Updates to third-party packages (EC-14086)
In this release the Cluster Manager is updated to use Apache 2.4.48, PHP 7.4.23, and OpenSSL 1.1.1l.
- Android support improvements
This release extends Android support to include builds based on AOSP 12 (EC-14110). For best results, users should regenerate their history files by running a full build with the
--emake-history=createoption; subsequent builds can be run without that option or with the value set to
merge. Additional fixes for AOSP include:
Incremental Android builds were sometimes slower than was necessary (EC-13515).
Some invocations of metalava were not being properly detected by jobcache (EC-14068).
- Yocto and Bitbake improvements
This release also includes enhancements to Bitbake and Yocto support to expand compatibility and improve preformance. The changes include:
Yocto 3.3 ("hardknott") is now supported (EC-14116).
Several packages, including linux-yocto, are now built with
Support is added for the following platforms:
CentOS 8 Stream (EC-14105)
Debian 10.8 (EC-14103)
Debian 10.9 (EC-14104)
Debian 10.10 (EC-14104)
Red Hat Enterprise Linux 8.4 (EC-14126)
Documentation: CloudBees Build Acceleration supported platforms
No database support changes.
No browser support changes.
- Jobcache uses cached file even when the source file has changed (EC-14117)
A bug in the C++ normalizing hash was preventing eMake’s jobcache functionality from detecting changes in source files which contained a C-style ending comment delimiter with more than one asterisk (e.g.,
- Improper handling of targets which are both intermediate and phony (EC-14106, EC-14109)
Previously, eMake did not handle targets which were marked both intermediate and phony properly, resulting in build failures where this occurred. The most common way this would arise is with the use of a
.SECONDARY:special target. This was preventing eMake from building the Linux kernel version 5.x (resulting in either hangs or with file-not-found build errors depending on the target architecture).
- emake build on Windows with many registry operations aborting intermittently (EC-14113, EC-14122)
Improper locking of registry annotations and update operations were causing intermittent eMake aborts, resulting in incomplete builds.
- emake does not support
MAKE_TERMERRenvironment variables (EC-12072, EC-14111)
Previously, eMake did not set these environment variables (when either stdout or stderr, respectively, is a tty). These are now set as appropriate.
- Agent utilization graph in Cluster Manager was showing fractional values (EC-14055)
The graph in the Cluster Manager’s Build Status > Performance > Agent utilization was showing fractional values on the y-axis.
- Unnecessary ledger update overhead when writing to an XFS file system (EC-14068)
An XFS file system (the default on CentOS 8.x and RHEL) would not be detected as being a local filesystem for updating the ledger, resulting in an unnecessary file copy.
- Add workaround for glibc bug causing deadlocks in builds with heavy ledger operations (EC-14119)
To avoid a deadlock in some versions of glibc, a workaround was added, and can be enabled by adding the
--emake-try-chain-locks=0command line option. See the "Known issues" section below for details.
- Cloud bursting
Because the agent to Cluster Manager communication uses TLS by default, the agent image used for AWS EC2, GCP or Azure cloud bursting must also be upgraded to the 11.3 release or later in order for cloud bursting to work.
For details about the new release strategy, see the CloudBees Build Acceleration Release Strategy Update web page.
- Version enforcement in Cluster Manager licensing
As of version 11.1, perpetual license files include a
Versionfield and are therefore no longer transferable to succeeding product versions. At run time, to be considered valid by the Cluster Manager, the license file must have a
Versionthat equals or exceeds the version of the Cluster Manager itself. Version checks consider only the
major.minorversion (for example, a license for 11.1 is valid for 11.1.0, 11.1.1, 11.1.2, and so on).
eMake authenticates with the Cluster Manager via the eMake/Cluster Manager protocol version to ensure that the eMake and Cluster Manager licenses match. If the license has no
Version field, it will be considered invalid by the 11.1 Cluster Manager.
You can upgrade just the Cluster Manager, but starting with version 11.1, if you upgrade eMake or agents, you must also upgrade the Cluster Manager. When you upgrade the Cluster Manager, you must also acquire a new license if you are currently using a perpetual license.
You can upgrade to new patch releases (but not new feature releases) without acquiring a new license. For example, if your current license specifies
11.1, then you can use a Cluster Manager with version
11.1.2, and so on. Also, for example, if your license specifies
12.0, then you can use a Cluster Manager with version
12.0.2, and so on, as well as
The Administration > Licenses page in the Cluster Manager web UI has a Version column, which provides the product version of each license in the list. This information is blank for a license with no explicit version (such as an evaluation license).
Make sure that you import a compliant license into the Cluster Manager before upgrading. For assistance, see CloudBees Support. For details about importing a license, see Logging In and Enabling Licensing - Importing Your License. (EC-13170 and EC-13154)
- Hardware requirements
The recommended total amount of RAM for an agent host is 2 GB per agent plus the amount of RAM normally needed to execute your build. For example, if you are running four agents, and your build normally needs 16 GB, you will need ((2 * 4) + 16) = 24 GB.
- Backing up before you upgrade
The upgrade process does not preserve the existing files. Back up the
/opt/ecloud/<arch>/clouddirectory for Linux or the
C:\ECloud\<arch>folder for Windows to a safe location.
For additional security, back up the database by following the recommended procedure from your database vendor.
- Installing JDBC drivers for MySQL or Oracle databases
CloudBees no longer distributes the JDBC drivers for MySQL or Oracle databases. To use one of these databases, you must download its driver directly from the Oracle website, then copy it to the appropriate directory on the Cluster Manager server, and then restart the Cluster Manager service. For details, see Installing JDBC Drivers for MySQL or Oracle Databases.
- If you relocate eMake
If you copy the
emakeexecutable to a new location, you must also copy the
execserverexecutable to that location. By default, the path to the
/opt/ecloud/i686_Linux/32/bin/execserver(for 32-bit eMake) or
/opt/ecloud/i686_Linux/64/bin/execserver(for 64-bit eMake).
- Regenerating history files after an upgrade
The identifier that is used to find certain types of jobs in the eMake history file changed in version 8.0. After an upgrade from version 7.2.2 or older versions to version 8.0 or newer versions, users should regenerate their history files by running their first build with the
--emake-history=createoption to avoid unnecessary serializations. This build might have more conflicts than normal (but subsequent builds should return to normal).
- Concurrent build licensing
As of version 9.1, for new CloudBees Build Acceleration subscription licenses, the number of builds that you can run concurrently is license-limited. The
noLicenseWaitTimeperformance metric indicates the amount of time that a build spent waiting for a concurrent build license because the number of concurrent builds reached the license limit. Also, as of version 9.1, JobCache is not separately licensed and is now included with the concurrent build license.
Customers using pre-9.1 CloudBees Build Acceleration licenses may continue to use those licenses, including the licenses for the JobCache add-on.
For details about licensing for concurrent builds, see the Logging In and Enabling Licensing. (EC-12095)
- Linux kernel issue that affects CloudBees Build Acceleration performance
Affected kernel versions:
RHEL kernel versions later than 2.6.18-194.32 and earlier than 2.6.32-131
Ubuntu Linux kernel versions 2.6.31, 2.6.32, 2.6.33, and 2.6.34
Affected systems might encounter reduced performance on both ext3 and ext4 file systems. Symptoms might include
hung_task_timeout_secsmessages in system dmesg logs
Widely variable agent availability (entering and exiting agent “penalty” status frequently)
Contention over the
Slower builds (with unexplained variances)
To help determine if this issue exists, run the
dmesg | grep hung_task_timeout command.
hung_task_timeout errors show that this issue is present. Contact your kernel provider for another version of the precompiled kernel.
- Deadlock hang in eMake client using ledger due to glibc on SLES 15.2 and 15.3
Some versions of glibc — releases roughly in the range 2.26 through 2.30, such as that present in SLES 15 SP2 and SP3 — have a bug in the try read-write lock pthread call that can result in deadlocks. This can cause builds with heavy ledger usage to intermittently hang in the eMake client. As a workaround for this, add the command line option
--emake-try-chain-locks=0to your build, which will avoid the try lock calls, though this will render the tracking of time waiting for locks in the resulting annotation file invalid. Other options would be upgrading your version of glibc, or switching to another distribution.
- Fixes for systems running RHEL 5.6, 5.7, 5.8, and 6.0
You should consider upgrading to 2.6.32-131 (RHEL 6.1) or downgrading to 2.6.18-194.32 (RHEL 5.5).
- Other known issues
Android M (6) is not compatible with CloudBees Build Acceleration 11.1. You must use Android M with CloudBees Build Acceleration 10.0.
If you kill a build manually, and the agents running on an Amazon EC2 instance fail to connect, the instance will continue to run. You must kill the instance manually.
If the Cluster Manager goes down while agents are running on an Amazon EC2 instance, they will still connect but will not be associated with a resource. You must kill the corresponding instance manually.
importDatacommand does not import license properties (such as
maxAgents). To work around this issue, re-import the license after using
You cannot control breakpoints from the Cluster Manager. (EC-12322)
If Apache fails to start properly after a new Cluster Manager installation, reboot the system.