CloudBees Build Acceleration 2020.10.00 preview release notes

8 minute read

CloudBees is pleased to announce the availability of the CloudBees Build Acceleration 2020.10.00 preview release.

New features and functionality

Redesigned and expanded Build Details

The Build Details page in the cluster manager has been redesigned and expanded to include performance analysis and visualizations derived from CloudBees Build Acceleration Insight. This enhancement enables users and administrators to easily access the most critical performance data for builds from any system that can access the cluster manager web interface, even if the annotation file for the build is not accessible. The Build Details page now includes several new sub-tabs:

  • Timeline: a visualization of the execution of jobs in the build.

  • Composition: analysis of the type of work performed in the build, such as compilation or packaging, and the portion of the total build workload represented by each.

  • Jobcache: analysis of the use and performance of jobcache in the build, including the cache hit rate and estimated time savings due to jobcache.

  • Performance: general performance information, including the longest serial chain in the build, if available.

  • Diagnostics: warnings and errors emitted by the build, determined by intelligent analysis of the build output log.

Android 11 support

This release extends the Android build integration to support builds based on AOSP 11.0.0_r3.

Bitbake support

This release extends Bitbake integration with support for packages that build using ninja instead of make, resulting in improved performance for Bitbake/Yocto users (EC-13789). In addition, the bb2anno utility has been enhanced to aggregate the data in <metrics> sections of the individual annotation files that are incorporated into the combined annotation file, to improve "whole build" performance analysis (EC-13956).

Security configuration

This release adds support for configuring agent security settings during installation and after installation with ecconfig. Security settings configured using these methods will be preserved during future upgrades. Prior to this change, configuring agent security settings required modification to the agent startup scripts, and those settings were not retained during upgrades (EC-13407).

Automatic output directory creation

This release introduces a new command-line option for emake, --emake-create-output-dirs. When enabled, this will cause emake to automatically and efficiently create output directories for targets when running in GMake emulation modes. In addition, the details of the implementation are such that this will avoid provoking conflicts even in builds that have no history file or have an incomplete history file. This feature is a superior alternative to mechanisms such as mkdir -p $(dirname $@), directory creation sentry files, and others that have commonly been employed with make-based build systems to solve the problem of creating output directories for build targets (EC-13680).

Build signatures and totality

This release adds "build signatures", a hash computed from the serial ordered list of output targets in the build. This value can be used to identify builds that are substantively "the same" to facilitate comparisons and identification of trends across sets of like builds. The hash uses emake-root-relative target names, so the signature will be the same even if the absolute location of the build has changed (for example, as in continuous integration builds, or between developers working on the same project from their own copies of the source tree). The build signature for a build can be found in the annotation properties.

This release also adds a new "totality" metric, which is the percentage of jobs that were not up-to-date in the build. For a full, from-scratch build, this metric should be 100 in most builds. For a "no touch" build, this metric should be close to zero. For incremental builds, the value will be somewhere between those values, depending on the portion of the output that was rebuilt.

CloudBees Build Acceleration Insight

This release includes an updated preview release of CloudBees Build Acceleration Insight with the following enhancements:

  • A new JobCache Summary report which provides an overview of jobcache use in the build including an estimate of the elapsed time saved and the types of jobcache used. (EI-845).

  • The Diagnostics report includes module and target name information in the command-line output (EI-853), and limits the number of results printed in both the GUI and command-line versions (EI-852).

In addition this release includes a variant of CloudBees Build Acceleration Insight specifically for command-line use on Linux, called einsight-cmd. This variant has no dependencies on GUI libraries, making it ideal for use in "headless" automation contexts.

Platform support

There are no changes in supported platforms in this release.

Behavior changes

Build comments are no longer shown on the Build Details page. This functionality is not widely used, so it was removed from the UI. Build comments are still accessible via cmtool.

The minimal annotation file uploaded unconditionally to the cluster manager at the end of a build no longer includes the <profile> tag, as it can be quite verbose on builds that used many agents.

Database support changes

No changes.

Browser support changes

No changes.

Resolved issues

bb2anno tool generates corrupted annotation (EC-13954)

If the annotation file for a task did not include at least waiting annotation detail and the annotation included recursive make invocations, bb2anno would erroneously truncate the task annotation data at the end of the first recursive make when incorporating it into the combined output annotation. With this fix, bb2anno instead detects task annotation files that do not have waiting annotation detail, prints a warning, and excludes those annotation files from the combined annotation.

None.

Other issues

None.

Installation and upgrade notes

Cloud bursting

Because the agent to Cluster Manager communication uses TLS by default, the agent image used for AWS EC2, GCP or Azure cloud bursting must also be upgraded to the 11.3 release or later in order for cloud bursting to work.

New CloudBees Build Acceleration release strategy

Starting in August 2019, the release strategy for CloudBees Build Acceleration is updated to add a “preview” release in addition to the standard long-term support (LTS), maintenance (patch), and hotfix releases.

The release numbering for the preview releases uses a new <year>.<month>.00 numbering scheme. For example, 2020.08.00.

CloudBees recommends that you upgrade to the preview release and test these features in a controlled environment before rolling them out to production.

For details about the new release strategy, see the CloudBees maintenance lifecycle policies web page.

Version enforcement in Cluster Manager licensing

As of version 11.1, perpetual license files include a Version field and are therefore no longer transferable to succeeding product versions. At run time, to be considered valid by the Cluster Manager, the license file must have a Version that equals or exceeds the version of the Cluster Manager itself. Version checks consider only the major.minor version (for example, a license for 11.1 is valid for 11.1.0, 11.1.1, 11.1.2, and so on).

eMake authenticates with the Cluster Manager via the eMake/Cluster Manager protocol version to ensure that the eMake and Cluster Manager licenses match. If the license has no Version field, it will be considered invalid by the 11.1 Cluster Manager.

You can upgrade just the Cluster Manager, but starting with version 11.1, if you upgrade eMake or agents, you must also upgrade the Cluster Manager. When you upgrade the Cluster Manager, you must also acquire a new license if you are currently using a perpetual license.

You can upgrade to new patch releases (but not new feature releases) without acquiring a new license. For example, if your current license specifies 11.1, then you can use a Cluster Manager with version 11.1, 11.1.1, 11.1.2, and so on. Also, for example, if your license specifies 12.0, then you can use a Cluster Manager with version 12.0, 12.0.1, 12.0.2, and so on, as well as 11.x.

The Administration > Licenses page in the Cluster Manager web UI has a Version column, which provides the product version of each license in the list. This information is blank for a license with no explicit version (such as an evaluation license).

Make sure that you import a compliant license into the Cluster Manager before upgrading. For assistance, see CloudBees Support. For details about importing a license, see Logging In and Enabling Licensing - Importing Your License. (EC-13170 and EC-13154)

Hardware requirements

  • The recommended total amount of RAM for an agent host is 2 GB per agent plus the amount of RAM normally needed to execute your build. For example, if you are running four agents, and your build normally needs 16 GB, you will need ((2 * 4) + 16) = 24 GB.

Backing up before you upgrade

  • The upgrade process does not preserve the existing files. Back up the /opt/ecloud/<arch>/cloud directory for Linux or the C:\ECloud\<arch> folder for Windows to a safe location.

  • For additional security, back up the database by following the recommended procedure from your database vendor.

Installing JDBC drivers for MySQL or Oracle databases

CloudBees no longer distributes the JDBC drivers for MySQL or Oracle databases. To use one of these databases, you must download its driver directly from the Oracle website, then copy it to the appropriate directory on the Cluster Manager server, and then restart the Cluster Manager service. For details, see Installing JDBC Drivers for MySQL or Oracle Databases.

If you relocate eMake

If you copy the emake executable to a new location, you must also copy the execserver executable to that location. By default, the path to the execserver executable is /opt/ecloud/i686_Linux/32/bin/execserver (for 32-bit eMake) or /opt/ecloud/i686_Linux/64/bin/execserver (for 64-bit eMake).

Regenerating history files after an upgrade

The identifier that is used to find certain types of jobs in the eMake history file changed in version 8.0. After an upgrade from version 7.2.2 or older versions to version 8.0or newer versions, users should regenerate their history files by running their first build with the --emake-history=create option to avoid unnecessary serializations. This build might have more conflicts than normal (but subsequent builds should return to normal).

Concurrent build licensing

As of version 9.1, for new CloudBees Build Acceleration subscription licenses, the number of builds that you can run concurrently is license-limited. The noLicenseWaitTime performance metric indicates the amount of time that a build spent waiting for a concurrent build license because the number of concurrent builds reached the license limit. Also, as of version 9.1, JobCache is not separately licensed and is now included with the concurrent build license.

Customers using pre-9.1 CloudBees Build Acceleration licenses may continue to use those licenses, including the licenses for the JobCache add-on.

For details about licensing for concurrent builds, see the Logging In and Enabling Licensing. (EC-12095)

Known issues

Linux kernel issue that affects CloudBees Build Acceleration performance

Affected kernel versions

  • RHEL kernel versions later than 2.6.18-194.32 and earlier than 2.6.32-131

  • Ubuntu Linux kernel versions 2.6.31, 2.6.32, 2.6.33, and 2.6.34

Symptoms

Affected systems might encounter reduced performance on both ext3 and ext4 file systems. Symptoms might include

  • hung_task_timeout_secs messages in system dmesg logs

  • Widely variable agent availability (entering and exiting agent “penalty” status frequently)

  • Contention over the ecagent.state file

  • Slower builds (with unexplained variances)

To help determine if this issue exists, run the dmesg | grep hung_task_timeout command. hung_task_timeout errors show that this issue is present. Contact your kernel provider for another version of the precompiled kernel.

Fixes for systems running RHEL 5.6, 5.7, 5.8, and 6.0

You should consider upgrading to 2.6.32-131 (RHEL 6.1) or downgrading to 2.6.18-194.32 (RHEL 5.5).

Other known issues

  • Android M is not compatible with CloudBees Build Acceleration 11.1. You must use Android M with CloudBees Build Acceleration 10.0.

  • If you kill a build manually, and the agents running on an Amazon EC2 instance fail to connect, the instance will continue to run. You must kill the instance manually.

  • If the Cluster Manager goes down while agents are running on an Amazon EC2 instance, they will still connect but will not be associated with a resource. You must kill the corresponding instance manually.

  • The cmtool importData command does not import license properties (such as maxAgents ). To work around this issue, re-import the license after using importData. (EC-12371)

  • You cannot control breakpoints from the Cluster Manager. (EC-12322)

  • If Apache fails to start properly after a new Cluster Manager installation, reboot the system.

Documentation

Product documentation

CloudBees Build Acceleration documentation for this preview release is available at the CloudBees Build Acceleration documentation page.

Documentation on the website is updated periodically.

Cluster Manager Online Help and Tooltips

Built into the Cluster Manager are:

  • A complete, robust, context-sensitive online help system (click the Help button in any page of the Cluster Manager web UI). See Web Interface Help for the latest updates to this information.

  • Tooltips with information to help fill in form fields.

Troubleshooting and Getting Help

Contacting CloudBees Support

To contact CloudBees Support, go to https://support.cloudbees.com/, and then click Submit a request to submit or see your support tickets.

Be prepared to provide your:

  • Name, title, company name, phone number, and email address

  • Operating system and version number

  • Product name and release version

  • Problem description

CloudBees Build Acceleration Knowledge Base

Go to https://support.cloudbees.com/hc/en-us/sections/360006494811 to find in-depth explanations of specific topics and solutions for specific problems.