CloudBees Build Acceleration 2020.09.00 preview release notes

8 minute read

CloudBees is pleased to announce the availability of the CloudBees Build Acceleration 2020.09.00 preview release.

New features and functionality

Performance improvements

This release of CloudBees Build Acceleration includes several enhancements to improve Android incremental build performance. For best results, users should regenerate their history files by running a full build with the --emake-history=create option; subsequent builds can be run without that option or with the value set to merge. The changes made include:

  • The ninja parser no longer creates jobs for prerequisites that are mentioned only in .ninja_deps. This avoids the cost of job processing for source files. (EC-13689)

  • emake no longer stat’s order-only prerequisites during up-to-date checks. (EC-13923)

  • The Android integration pragmafiles (enabled when you use the --emake-android-version option) have been retuned for Android 10 and Android 11 to avoid spurious rebuilds of up-to-date outputs during incremental builds and to reduce startup time. (EC-13916, EC-13915)

In addition, this release includes an enhancement to the way symbolic links are handled, such that timestamps on symlinks created during the build are in the correct relative order when compared to timestamps on regular files created during the build. This avoids spurious rebuilds of up-to-date outputs during incremental builds. (EC-13444)

CloudBees Build Acceleration Insight

This release includes an updated preview release of CloudBees Build Acceleration Insight with the following enhancements:

  • A new "Diagnostics" report identifies error and warning messages in the build output log and has both graphical and command-line interfaces. (EI-843)

  • The "Job Time by Type" report can produce JSON formatted output when invoked from the command-line by adding a "json" parameter to the invocation. For example: einsight --report="JobTimeByType json" emake.xml. (EI-848)

  • The "Longest Serial Chain" report can produce JSON formatted output when invoked from the command-line by adding a "json" parameter to the invocation. For example, to generate the longest serial chain overall in json format, use einsight --report="LongestSerialChain overall json" emake.xml. (EI-844)

Platform support

There are no changes in supported platforms in this release.

Documentation: CloudBees Build Acceleration supported platforms

Behavior changes

As part of CloudBees' ongoing commitment to inclusivity, the bemake configuration files used to control which bitbake packages should be built using emake and which should be built using gmake have been renamed to bemake_package_accelerated and bemake_package_exceptions, respectively. For backwards compatibility the legacy names bemake_package_whitelist and bemake_package_blacklist will be used if files matching the new names cannot be found, but the legacy names are considered deprecated and anybody using CloudBees Build Acceleration to build using bitbake is encouraged to rename these files in their setup, if present. (EC-13892)

Database support changes

No changes.

Browser support changes

No changes.

Resolved issues

emake assertion over willConflict flag (EC-13932)

With this fix, instead of failing with an assertion, emake will treat a job as in conflict if previous job activity indicated that the job should be in conflict, even if the file usage at the end of the job would not otherwise indicate a conflict.

Error when upgrading auto-scaling cloudburst agents (EC-13880)

With this fix, the installer correctly handles upgrades of agent hosts that are configured to automatically select the number of agents at startup, as is typical for cloudburst agent machine images.

Jobcached jobs do not update autodep data (EC-13924)

With this fix, emake correctly updates autodep information based on usage from jobs pulled from the jobcache.

Not possible to clear autodep data for a job (EC-13925)

Prior to this change there was no way to completely expunge the autodep information for a single job, even if the build was changed such that autodep no longer applied to the job. With this fix, emake will clear the autodep entry for the job if it is no longer applicable.

Build hangs waiting for cloudburst agents (EC-13805)

A race condition sometimes caused the cluster manager to fail to remove the database entries for agents corresponding to cloudburst instances that had been terminated. With this fix, the race condition is eliminated so the database is updated reliably.

Cloudburst agent instances are not always terminated automatically (EC-13072)

If the cluster manager was restarted while cloudburst agents were active, the cluster manager would lose track of the agents after the restart. The cluster manager would subsequently fail to terminate those agent instances, and would fail to reuse the agents for new builds. With this fix, the cluster manager correctly resumes management of cloudburst agents after restarting, ensuring that the agents are reused when appropriate and that the agents are terminated as expected.

gobject-introspection do_compile task fails when building Yocto with ebitbake (EC-13918)

The gobject-introspection package build is not compatible with emake. This fix modifies the default Yocto integration to use GNU make to build gobject-introspection instead.

Security-related issues

None.

Other issues

None.

Installation and upgrade notes

Cloud bursting

Because the agent to Cluster Manager communication uses TLS by default, the agent image used for AWS EC2, GCP or Azure cloud bursting must also be upgraded to the 11.3 release or later in order for cloud bursting to work.

New CloudBees Build Acceleration release strategy

Starting in August 2019, the release strategy for CloudBees Build Acceleration is updated to add a “preview” release in addition to the standard long-term support (LTS), maintenance (patch), and hotfix releases.

The release numbering for the preview releases uses a new <year>.<month>.00 numbering scheme. For example, 2020.08.00.

CloudBees recommends that you upgrade to the preview release and test these features in a controlled environment before rolling them out to production.

For details about the new release strategy, see the CloudBees maintenance lifecycle policies web page.

Version enforcement in Cluster Manager licensing

As of version 11.1, perpetual license files include a Version field and are therefore no longer transferable to succeeding product versions. At run time, to be considered valid by the Cluster Manager, the license file must have a Version that equals or exceeds the version of the Cluster Manager itself. Version checks consider only the major.minor version (for example, a license for 11.1 is valid for 11.1.0, 11.1.1, 11.1.2, and so on).

eMake authenticates with the Cluster Manager via the eMake/Cluster Manager protocol version to ensure that the eMake and Cluster Manager licenses match. If the license has no Version field, it will be considered invalid by the 11.1 Cluster Manager.

You can upgrade just the Cluster Manager, but starting with version 11.1, if you upgrade eMake or agents, you must also upgrade the Cluster Manager. When you upgrade the Cluster Manager, you must also acquire a new license if you are currently using a perpetual license.

You can upgrade to new patch releases (but not new feature releases) without acquiring a new license. For example, if your current license specifies 11.1, then you can use a Cluster Manager with version 11.1, 11.1.1, 11.1.2, and so on. Also, for example, if your license specifies 12.0, then you can use a Cluster Manager with version 12.0, 12.0.1, 12.0.2, and so on, as well as 11.x.

The Administration > Licenses page in the Cluster Manager web UI has a Version column, which provides the product version of each license in the list. This information is blank for a license with no explicit version (such as an evaluation license).

Make sure that you import a compliant license into the Cluster Manager before upgrading. For assistance, see CloudBees Support. For details about importing a license, see Logging In and Enabling Licensing - Importing Your License. (EC-13170 and EC-13154)

Hardware requirements

  • The recommended total amount of RAM for an agent host is 2 GB per agent plus the amount of RAM normally needed to execute your build. For example, if you are running four agents, and your build normally needs 16 GB, you will need ((2 * 4) + 16) = 24 GB.

Backing up before you upgrade

  • The upgrade process does not preserve the existing files. Back up the /opt/ecloud/<arch>/cloud directory for Linux or the C:\ECloud\<arch> folder for Windows to a safe location.

  • For additional security, back up the database by following the recommended procedure from your database vendor.

Installing JDBC drivers for MySQL or Oracle databases

CloudBees no longer distributes the JDBC drivers for MySQL or Oracle databases. To use one of these databases, you must download its driver directly from the Oracle website, then copy it to the appropriate directory on the Cluster Manager server, and then restart the Cluster Manager service. For details, see Installing JDBC Drivers for MySQL or Oracle Databases.

If you relocate eMake

If you copy the emake executable to a new location, you must also copy the execserver executable to that location. By default, the path to the execserver executable is /opt/ecloud/i686_Linux/32/bin/execserver (for 32-bit eMake) or /opt/ecloud/i686_Linux/64/bin/execserver (for 64-bit eMake).

Regenerating history files after an upgrade

The identifier that is used to find certain types of jobs in the eMake history file changed in version 8.0. After an upgrade from version 7.2.2 or older versions to version 8.0or newer versions, users should regenerate their history files by running their first build with the --emake-history=create option to avoid unnecessary serializations. This build might have more conflicts than normal (but subsequent builds should return to normal).

Concurrent build licensing

As of version 9.1, for new CloudBees Build Acceleration subscription licenses, the number of builds that you can run concurrently is license-limited. The noLicenseWaitTime performance metric indicates the amount of time that a build spent waiting for a concurrent build license because the number of concurrent builds reached the license limit. Also, as of version 9.1, JobCache is not separately licensed and is now included with the concurrent build license.

Customers using pre-9.1 CloudBees Build Acceleration licenses may continue to use those licenses, including the licenses for the JobCache add-on.

For details about licensing for concurrent builds, see the Logging In and Enabling Licensing. (EC-12095)

Known issues

Linux kernel issue that affects CloudBees Build Acceleration performance

Affected kernel versions

  • RHEL kernel versions later than 2.6.18-194.32 and earlier than 2.6.32-131

  • Ubuntu Linux kernel versions 2.6.31, 2.6.32, 2.6.33, and 2.6.34

Symptoms

Affected systems might encounter reduced performance on both ext3 and ext4 file systems. Symptoms might include

  • hung_task_timeout_secs messages in system dmesg logs

  • Widely variable agent availability (entering and exiting agent “penalty” status frequently)

  • Contention over the ecagent.state file

  • Slower builds (with unexplained variances)

To help determine if this issue exists, run the dmesg | grep hung_task_timeout command. hung_task_timeout errors show that this issue is present. Contact your kernel provider for another version of the precompiled kernel.

Fixes for systems running RHEL 5.6, 5.7, 5.8, and 6.0

You should consider upgrading to 2.6.32-131 (RHEL 6.1) or downgrading to 2.6.18-194.32 (RHEL 5.5).

Other known issues

  • Android M is not compatible with CloudBees Build Acceleration 11.1. You must use Android M with CloudBees Build Acceleration 10.0.

  • If you kill a build manually, and the agents running on an Amazon EC2 instance fail to connect, the instance will continue to run. You must kill the instance manually.

  • If the Cluster Manager goes down while agents are running on an Amazon EC2 instance, they will still connect but will not be associated with a resource. You must kill the corresponding instance manually.

  • The cmtool importData command does not import license properties (such as maxAgents ). To work around this issue, re-import the license after using importData. (EC-12371)

  • You cannot control breakpoints from the Cluster Manager. (EC-12322)

  • If Apache fails to start properly after a new Cluster Manager installation, reboot the system.

Documentation

Product documentation

CloudBees Build Acceleration documentation for this preview release is available at the CloudBees Build Acceleration documentation page.

Documentation on the website is updated periodically.

Cluster Manager Online Help and Tooltips

Built into the Cluster Manager are:

  • A complete, robust, context-sensitive online help system (click the Help button in any page of the Cluster Manager web UI). See Web Interface Help for the latest updates to this information.

  • Tooltips with information to help fill in form fields.

Troubleshooting and Getting Help

Contacting CloudBees Support

To contact CloudBees Support, go to https://support.cloudbees.com/, and then click Submit a request to submit or see your support tickets.

Be prepared to provide your:

  • Name, title, company name, phone number, and email address

  • Operating system and version number

  • Product name and release version

  • Problem description

CloudBees Build Acceleration Knowledge Base

Go to https://support.cloudbees.com/hc/en-us/sections/360006494811 to find in-depth explanations of specific topics and solutions for specific problems.