CloudBees is pleased to announce the availability of the CloudBees Build Acceleration 12.0 long-term support release.
CloudBees Build Acceleration is a software build accelerator that dramatically reduces build times by distributing the build over a large cluster of inexpensive servers. CloudBees Build Acceleration uses a patented dependency-management system to identify and fix problems in real time that break traditional parallel builds. CloudBees Build Acceleration plugs seamlessly into existing software development environments and includes web-based management and reporting tools.
CloudBees Build Acceleration includes the following components:
-
Electric Make (“eMake”)
-
Electric File System (EFS)
-
CloudBees Build Acceleration agents (“Agents”)
-
Cluster Manager
-
Electrify
Security fixes
- Updates to third-party packages (EC-14004, EC-14049)
-
In this release the Cluster Manager is updated to use Apache 2.4.46, PHP 7.4.12, and OpenSSL 1.1.1i.
New features
- Redesigned and expanded Build Details
-
The Build Details page in the Cluster Manager has been redesigned and expanded to include performance analysis and visualizations derived from CloudBees Build Acceleration Insight. This enhancement enables users and administrators to easily access the most critical performance data for builds from any system that can access the Cluster Manager web interface, even if the annotation file for the build is not accessible. The Build Details page now includes several new sub-tabs:
-
Timeline: a visualization of the execution of jobs in the build.
-
Composition: analysis of the type of work performed in the build, such as compilation or packaging, and the portion of the total build workload represented by each.
-
Jobcache: analysis of the use and performance of jobcache in the build, including the cache hit rate and estimated time savings due to jobcache.
-
Performance: general performance information, including the number of agents in use over the lifetime of the build and a prioritized summary of the build critical path so users can quickly identify those jobs that have the greatest impact on the build duration.
-
Diagnostics: warnings and errors emitted by the build, determined by intelligent analysis of the build output log.
-
Recommendations: actionable advice for improving build performance based on analysis of the build and identification of several known performance anti-patterns, such as missing or incomplete history, jobcache misses and more. Recommendations are prioritized according to the estimated impact on performance.
-
- Other Cluster Manager enhancements
-
In addition to the updates described above, the Cluster Manager includes a number of other improvements:
-
For running builds, the Builds and Build Details pages in the Cluster Manager have been enhanced to show the build percentage complete.
-
When editing Resources, the Hosts text box is now a pull down with known agent hosts, to simplify the task of defining resources.
-
A Description column has been added to the Resources page.
-
Users can specify a different number of maximum items on the Builds and Agents pages.
-
- Android support improvements
-
This release extends Android support to include builds based on AOSP 11. In addition, this release features several performance improvements for both full and incremental Android builds. For best results, users should regenerate their history files by running a full build with the
--emake-history=create
option; subsequent builds can be run without that option or with the value set tomerge
. The changes made include:-
The
javac
jobcache type will now apply to jobs that usekotlinc
and known aliases for the Kotlin compiler. (EC-13917) -
A custom hash was introduced for Unix archive files which ignores member timestamps. This allows for cache hits when the objects in the archive are unchanged except for their modification time. (EC-14020)
-
A custom hash was introduced for C/C++ source files and headers which ignores comments and blank lines. This allows for cache hits when the code in source files is unchanged even if the comments are different. This is especially beneficial for generated source files, which frequently embed the time of generation in a comment. (EC-11533)
-
The jobcache feature was modified to omit explicit timestamp modification from the cached operations, so that outputs extracted from the cache will be assigned a current time stamp when they are created. This avoids unexpected rebuilds in incremental builds after full builds that used the jobcache. (EC-13699)
-
The ninja parser no longer creates jobs for prerequisites that are mentioned only in
.ninja_deps
. This avoids the cost of job processing for source files. (EC-13689) -
emake no longer stat’s order-only prerequisites during up-to-date checks. (EC-13923)
-
The Android integration pragmafiles (enabled when you use the
--emake-android-version
option) have been retuned for Android 10 and Android 11 to avoid spurious rebuilds of up-to-date outputs during incremental builds and to reduce startup time. (EC-13916, EC-13915)
-
- Bitbake and Yocto improvements
-
This release also includes enhancements to Bitbake and Yocto support to expand compatibility and improve preformance. The changes include:
-
Yocto 3.2 ("gatesgarth") is now supported.
-
Parse avoidance is enabled by default and
emake
is used to build more packages. -
The integration now supports building packages that use
ninja
instead ofmake
, via the ninja emulation mode inemake
. The default configuration is updated to take advantage of this change, resulting in improved performance for Bitbake/Yocto users. (EC-13879) -
The
bb2anno
utility has been enhanced to aggregate the data in<metrics>
sections of the individual annotation files that are incorporated into the combined annotation file, to improve "whole build" performance analysis. (EC-13956)
-
- Security configuration
-
This release adds support for configuring agent security settings during installation and after installation with
ecconfig
. Security settings configured using these methods will be preserved during future upgrades. Prior to this change, configuring agent security settings required modification to the agent startup scripts, and those settings were not retained during upgrades (EC-13407). - Automatic output directory creation
-
This release introduces a new command-line option for emake,
--emake-create-output-dirs
. When enabled, this will cause emake to automatically and efficiently create output directories for targets when running in GMake emulation modes. In addition, the details of the implementation are such that this will avoid provoking conflicts even in builds that have no history file or have an incomplete history file. This feature is a superior alternative to mechanisms such asmkdir -p $(dirname $@)
, directory creation sentry files, and others that have commonly been employed with make-based build systems to solve the problem of creating output directories for build targets (EC-13680). - Build signatures and totality
-
This release adds "build signatures", a hash computed from the serial ordered list of output targets in the build. This value can be used to identify builds that are substantively "the same" to facilitate comparisons and identification of trends across sets of like builds. The hash uses emake-root-relative target names, so the signature will be the same even if the absolute location of the build has changed (for example, as in continuous integration builds, or between developers working on the same project from their own copies of the source tree). The build signature for a build can be found in the annotation properties.
This release also adds a new "totality" metric, which is the percentage of jobs that were not up-to-date in the build. For a full, from-scratch build, this metric should be 100 in most builds. For a "no touch" build, this metric should be close to zero. For incremental builds, the value will be somewhere between those values, depending on the portion of the output that was rebuilt.
- Insight updates
-
This release includes an updated release of CloudBees Build Acceleration Insight with several enhancements to support the new Build Details page in the Cluster Manager:
-
A new Recommendations report which analyzes build performance and provides recommendations of ways to improve performance. (EI-846)
-
A new JobCache Summary report which provides an overview of jobcache use in the build including an estimate of the elapsed time saved and the types of jobcache used. (EI-845)
-
A new Diagnostics report identifies error and warning messages in the build output log. (EI-843)
-
The Agent Utilization report now shows agent utilization over the lifetime of the build, instead of a histogram-like summary of utilization. This is both more intuitive and more useful since it allows the user to quickly see when the number of agents in use was low during the build.
-
In addition this release includes a variant of CloudBees Build Acceleration Insight specifically for command-line use on Linux, called einsight-cmd
. This variant has no dependencies on GUI libraries, making it ideal for use in "headless" automation contexts.
- Other improvements
-
This release includes an enhancement to the way symbolic links are handled, such that timestamps on symlinks created during the build are in the correct relative order when compared to timestamps on regular files created during the build. This avoids spurious rebuilds of up-to-date outputs during incremental builds. (EC-13444)
This release adds a new <configuration>
section to annotation files, which describes the actual values used for every user-configurable option in emake. This eliminates ambiguity about the settings used in each build by providing a clear record that can be referenced when debugging or during performance analysis.
New platform support
Support is added for the following platforms:
-
CentOS 8.3
-
CentOS 8.2
-
CentOS 7.9
-
Debian 10.7
-
Debian 10.6
-
Debian 10.5
-
Debian 9.13
-
Red Hat Enterprise Linux 8.3
-
Red Hat Enterprise Linux 7.9
-
SUSE Linux Enterprise Server 15.2
-
Ubuntu 20.10
-
Windows Server 2019
In addition, the following platforms are now deprecated: these platforms are supported in 12.0, but they will not be supported in future releases:
-
Red Hat Enterprise Linux 5.x
-
Red Hat Enterprise Linux 6.0 through 6.9
-
ClearCase 7 and 8
-
32-bit ClearCase 9
Furthermore, after the 12.0 release, electrify will not support monitoring 32-bit applications.
Finally, after the 12.0 release, the minimum browser requirement for the Cluster Manager web interface will be updated to the following versions:
-
Chrome 57
-
Edge 16
-
Firefox 52
-
Safari 11
Documentation: CloudBees Build Acceleration supported platforms
No database support changes.
No browser support changes.
Resolved issues
- emake assertion over willConflict flag (EC-13932)
-
With this fix, instead of failing with an assertion, emake will treat a job as in conflict if previous job activity indicated that the job should be in conflict, even if the file usage at the end of the job would not otherwise indicate a conflict.
- electrify does not escape regex characters in
--electrify-remote
option (EC-13097) -
With this fix, electrify treats the values specified in
--electrify-remote
as literal characters as expected, rather than as regex characters. This means that the user can specify--electrify-remote=g`, for example, and it will match the literal string `g
rather than "g repeated one or more times". Use--electrify-allow-regex
to specify process names using regex syntax. - Error when upgrading auto-scaling cloudburst agents (EC-13880)
-
With this fix, the installer correctly handles upgrades of agent hosts that are configured to automatically select the number of agents at startup, as is typical for cloudburst agent machine images.
- Jobcached jobs do not update autodep data (EC-13924)
-
With this fix, emake correctly updates autodep information based on usage from jobs pulled from the jobcache.
- Not possible to clear autodep data for a job (EC-13925)
-
Prior to this change there was no way to completely expunge the autodep information for a single job, even if the build was changed such that autodep no longer applied to the job. With this fix, emake will clear the autodep entry for the job if it is no longer applicable.
- Build hangs waiting for cloudburst agents (EC-13805)
-
A race condition sometimes caused the Cluster Manager to fail to remove the database entries for agents corresponding to cloudburst instances that had been terminated. With this fix, the race condition is eliminated so the database is updated reliably.
- Cloudburst agent instances are not always terminated automatically (EC-13072)
-
If the Cluster Manager was restarted while cloudburst agents were active, the Cluster Manager would lose track of the agents after the restart. The Cluster Manager would subsequently fail to terminate those agent instances, and would fail to reuse the agents for new builds. With this fix, the Cluster Manager correctly resumes management of cloudburst agents after restarting, ensuring that the agents are reused when appropriate and that the agents are terminated as expected.
- bb2anno tool generates corrupted annotation (EC-13954)
-
If the annotation file for a task did not include at least
waiting
annotation detail and the annotation included recursive make invocations,bb2anno
would erroneously truncate the task annotation data at the end of the first recursive make when incorporating it into the combined output annotation. With this fix,bb2anno
instead detects task annotation files that do not havewaiting
annotation detail, prints a warning, and excludes those annotation files from the combined annotation. - Deleting builds from the Cluster Manager deletes too much from
build_logs
directory (EC-13982) -
A defect in the Cluster Manager where deleting the logs for builds that were deleted via the Cluster Manager web interface or with
cmtool
resulted in the deletion of build logs for unrelated builds. With this fix, only the logs for the deleted builds are deleted. - Cluster Manager starts more agent containers than Kubernetes can sustain (EC-13660)
-
When using Kubernetes cloudbursting, the Cluster Manager failed to specify a CPU count when making agent container requests. This could result in the Kubernetes cluster trying to run more containers on a host than that host could reasonably support. With this fix, the Cluster Manager specifies a CPU count when starting new agent containers, enabling Kubernetes to correctly limit total deployment and balancing loads across hosts in the Kubernetes cluster.
Behavior changes
The jobcache version has been updated to version 34, as some of the enhancements in this release are incompatible with the previously recorded data in the cache. Builds using the new version will automatically repopulate the cache again from scratch. Users may wish to manually delete the old cache data to reclaim disk space.
With this release, emake no longer records explicit timestamp modifications, such as those arising from commands like touch -t …
in operations saved to jobcache slots. This is consistent with other build caching tools, and eliminates a class of problems relating to replaying old file timestamps in new builds (EC-13699).
As part of CloudBees' ongoing commitment to inclusivity, the bemake configuration files used to control which bitbake packages should be built using emake and which should be built using gmake have been renamed to bemake_package_accelerated
and bemake_package_exceptions
, respectively. For backwards compatibility the legacy names bemake_package_whitelist
and bemake_package_blacklist
will be used if files matching the new names cannot be found, but the legacy names are considered deprecated and anybody using CloudBees Build Acceleration to build using bitbake is encouraged to rename these files in their setup, if present. (EC-13892)
Installation notes
- Cloud bursting
-
Because the agent to Cluster Manager communication uses TLS by default, the agent image used for AWS EC2, GCP or Azure cloud bursting must also be upgraded to the 11.3 release or later in order for cloud bursting to work.
- New CloudBees Build Acceleration release strategy
-
Starting in August 2019, the release strategy for CloudBees Build Acceleration is updated to add a “preview” release in addition to the standard long-term support (LTS), maintenance (patch), and hotfix releases.
The release numbering for the preview releases uses a new <year>.<month>.00
numbering scheme. For example, 2020.08.00
.
CloudBees recommends that you upgrade to the preview release and test these features in a controlled environment before rolling them out to production. |
For details about the new release strategy, see the CloudBees maintenance lifecycle policies web page.
- Version enforcement in Cluster Manager licensing
-
As of version 11.1, perpetual license files include a
Version
field and are therefore no longer transferable to succeeding product versions. At run time, to be considered valid by the Cluster Manager, the license file must have aVersion
that equals or exceeds the version of the Cluster Manager itself. Version checks consider only themajor.minor
version (for example, a license for 11.1 is valid for 11.1.0, 11.1.1, 11.1.2, and so on).
eMake authenticates with the Cluster Manager via the eMake/Cluster Manager protocol version to ensure that the eMake and Cluster Manager licenses match. If the license has no Version
field, it will be considered invalid by the 11.1 Cluster Manager.
You can upgrade just the Cluster Manager, but starting with version 11.1, if you upgrade eMake or agents, you must also upgrade the Cluster Manager. When you upgrade the Cluster Manager, you must also acquire a new license if you are currently using a perpetual license.
You can upgrade to new patch releases (but not new feature releases) without acquiring a new license. For example, if your current license specifies 11.1
, then you can use a Cluster Manager with version 11.1
, 11.1.1
, 11.1.2
, and so on. Also, for example, if your license specifies 12.0
, then you can use a Cluster Manager with version 12.0
, 12.0.1
, 12.0.2
, and so on, as well as 11.x
.
The Administration > Licenses page in the Cluster Manager web UI has a Version column, which provides the product version of each license in the list. This information is blank for a license with no explicit version (such as an evaluation license).
Make sure that you import a compliant license into the Cluster Manager before upgrading. For assistance, see CloudBees Support. For details about importing a license, see Logging In and Enabling Licensing - Importing Your License. (EC-13170 and EC-13154)
- Hardware requirements
-
The recommended total amount of RAM for an agent host is 2 GB per agent plus the amount of RAM normally needed to execute your build. For example, if you are running four agents, and your build normally needs 16 GB, you will need ((2 * 4) + 16) = 24 GB.
- Backing up before you upgrade
-
The upgrade process does not preserve the existing files. Back up the
/opt/ecloud/<arch>/cloud
directory for Linux or theC:\ECloud\<arch>
folder for Windows to a safe location.
For additional security, back up the database by following the recommended procedure from your database vendor.
- Installing JDBC drivers for MySQL or Oracle databases
-
CloudBees no longer distributes the JDBC drivers for MySQL or Oracle databases. To use one of these databases, you must download its driver directly from the Oracle website, then copy it to the appropriate directory on the Cluster Manager server, and then restart the Cluster Manager service. For details, see Installing JDBC Drivers for MySQL or Oracle Databases.
- If you relocate eMake
-
If you copy the
emake
executable to a new location, you must also copy theexecserver
executable to that location. By default, the path to theexecserver
executable is/opt/ecloud/i686_Linux/32/bin/execserver
(for 32-bit eMake) or/opt/ecloud/i686_Linux/64/bin/execserver
(for 64-bit eMake). - Regenerating history files after an upgrade
-
The identifier that is used to find certain types of jobs in the eMake history file changed in version 8.0. After an upgrade from version 7.2.2 or older versions to version 8.0 or newer versions, users should regenerate their history files by running their first build with the
--emake-history=create
option to avoid unnecessary serializations. This build might have more conflicts than normal (but subsequent builds should return to normal). - Concurrent build licensing
-
As of version 9.1, for new CloudBees Build Acceleration subscription licenses, the number of builds that you can run concurrently is license-limited. The
noLicenseWaitTime
performance metric indicates the amount of time that a build spent waiting for a concurrent build license because the number of concurrent builds reached the license limit. Also, as of version 9.1, JobCache is not separately licensed and is now included with the concurrent build license.
Customers using pre-9.1 CloudBees Build Acceleration licenses may continue to use those licenses, including the licenses for the JobCache add-on.
For details about licensing for concurrent builds, see the Logging In and Enabling Licensing. (EC-12095)
Known issues
- Linux kernel issue that affects CloudBees Build Acceleration performance
-
Affected kernel versions:
-
RHEL kernel versions later than 2.6.18-194.32 and earlier than 2.6.32-131
-
Ubuntu Linux kernel versions 2.6.31, 2.6.32, 2.6.33, and 2.6.34
-
Symptoms:
Affected systems might encounter reduced performance on both ext3 and ext4 file systems. Symptoms might include
-
hung_task_timeout_secs
messages in system dmesg logs -
Widely variable agent availability (entering and exiting agent “penalty” status frequently)
-
Contention over the
ecagent.state
file -
Slower builds (with unexplained variances)
To help determine if this issue exists, run the dmesg | grep hung_task_timeout
command. hung_task_timeout
errors show that this issue is present. Contact your kernel provider for another version of the precompiled kernel.
- Fixes for systems running RHEL 5.6, 5.7, 5.8, and 6.0
-
You should consider upgrading to 2.6.32-131 (RHEL 6.1) or downgrading to 2.6.18-194.32 (RHEL 5.5).
- Other known issues
-
-
Android M is not compatible with CloudBees Build Acceleration 11.1. You must use Android M with CloudBees Build Acceleration 10.0.
-
If you kill a build manually, and the agents running on an Amazon EC2 instance fail to connect, the instance will continue to run. You must kill the instance manually.
-
If the Cluster Manager goes down while agents are running on an Amazon EC2 instance, they will still connect but will not be associated with a resource. You must kill the corresponding instance manually.
-
The cmtool
importData
command does not import license properties (such asmaxAgents
). To work around this issue, re-import the license after usingimportData
. (EC-12371) -
You cannot control breakpoints from the Cluster Manager. (EC-12322)
-
If Apache fails to start properly after a new Cluster Manager installation, reboot the system.
-