CloudBees Build Acceleration 11.3 release notes

Product description

CloudBees Build Acceleration is a software build accelerator that dramatically reduces build times by distributing the build over a large cluster of inexpensive servers. CloudBees Build Acceleration uses a patented dependency-management system to identify and fix problems in real time that break traditional parallel builds. CloudBees Build Acceleration plugs seamlessly into existing software development environments and includes web-based management and reporting tools.

CloudBees Build Acceleration includes the following components:

  • Electric Make (“eMake”)

  • Electric File System (EFS)

  • CloudBees Build Acceleration agents (“Agents”)

  • Cluster Manager

  • Electrify

What’s new or modified

Introducing CloudBees Build Acceleration

CloudBees is introducing new, self-describing product names across our entire product line that make them easier for anyone in our target market to find, and to understand intuitively what they do.

With the release of version 11.3, what you’ve known previously as CloudBees Accelerator, or even ElectricAccelerator, is now called CloudBees Build Acceleration. The only thing that’s changed is the name—​and the new features listed below.

Visual Studio Plugin and Insight have also been updated with the new name.

Documentation: Simulating builds

Performance improvements & support for the latest Android Open Source Project (AOSP) builds

  • Out-of-the-box support & optimized performance when building AOSP 11 is introduced.

  • Users can use the --emake-android-version=11.0.0 option to seamlessly accelerate the building of the AOSP v11 preview release with CloudBees Build Acceleration.

  • Support for emulating Ninja v1.9.0 which is a prerequisite for AOSP v11 and for modern CMake support. This enables CloudBees Build Acceleration to support the latest version and older versions of Android concurrently.

Performance improvements & Support for the latest Yocto open source project builds

  • This release introduces support for Yocto builds version 2.1, v3.0 & v3.1 and delivers improved performance out-of-the-box through the accelerated building of modules such as glibc. (EC-13426) (EC-13518)

  • Support for Yocto 3.1 (Dunfell) has been added to ebitbake.

Caching & scalability improvements

  • Jobcache support is added for .oS shared object files produced by the GCC compiler.

  • The responsiveness of large, busy clusters has been improved.

  • Electric File System (EFS) performance has been improved through improved memory management.

Google Cloud Cloud Bursting improvements with IAM, support for preemptible instances and VPC network configuration options

  • IAM role support for Google Cloud Platform (GCP) is introduced. When creating credentials on the Cluster Manager, you can now choose between using the service account file or using the application default credential by choosing the Server IAM role. (The Cluster Manager instance must be started with a service account that is configured with permissions to launch, list, and delete instances.) (EC-13734 and EC-13718)

  • Cloud bursting for Google Cloud Platform (GCP) now allows the user to specify the GCP VPC name of the network. If the GCP VPC network was created in custom mode, the subnetwork must also be specified. (EC-13750)

  • When creating a Google Cloud Platform (GCP) cloud bursting resource, selecting the Prefer Preemptible Instances checkbox causes the cloud bursting instances to be launched with preemptible scheduling.

Improved Security for emake to CM and agents to CM communication

  • By default CloudBees Build Acceleration will now use TLS communication between emake and the Cluster Manager and the agents and the Cluster Manager. When installing the Cluster Manager, the installer will create a self-signed certificate and the host name will be verified during the communication. Customers wishing to upgrade to trusted certificates can use the emake option --emake-cm-security=strict along with the --emake-ssl-cacert and --emake-cm-keystore options. The --emake-cm-security=none option can be used to turn off TLS communication.

  • On the agent side strict security can be turned on using the ecagent service script and modifying the -cmsecurity option. The --cacert and -cmkeystore options must also be provided.

Improved Gnu Make compatibility

  • The $(file …​) function is now supported for reading, writing, and appending for gmake 4.x. (EC-12065)

  • Support for gmake 3.81 and 3.82 is improved: If an -include directive references a target that has nonexistent prerequisites, as long as that target is not a goal, eMake will succeed. If it is a goal, eMake will fail with the appropriate error. (EC-6117)

Usability Improvements

Simulating builds with moxie

CloudBees Build Acceleration now includes moxie, a tool to test basic cluster setup and demonstrate acceleration features. Moxie works by generating a synthetic build-like workload, including makefiles, compilers, linkers and other tools typically found in a software build.

Real time visualization of agent allocation

The new "Real Time Visualization of Agent Allocation" report in the Cluster Manager visually shows the agent allocation in real time. Refer to Reports page - Real Time Visualization of Agent Allocation for details.

Emake usability improved

eMake now produces a warning if it discovers an output target which is not contained within in any of the eMake root set of directories.

Build details page

Several improvements to the Build Details page. (EC-13791)

  • Information grouping is more convenient.

  • Primary build information formatting improved.

  • Report selector is replaced by tabs.

Agent details page

Several improvements to the Agent Details page. (EC-13802)

Performance improvements

  • The responsiveness of large, busy clusters has been improved.

  • Electric File System (EFS) performance has been improved by using slab allocators for enodes.

Other new features and functionality

  • The Build Details pane in the Builds tab of the Cluster Manager web UI now includes a Speedup field, which shows the speedup (in "x-times" notation) relative to a non-accelerated build. The calculation of this value includes the speedup provided by JobCache if enabled. (EC-13675)

Behavior changes

  • The URLs for the context-sensitive online Help in the Cluster Manager web UI are updated to point to the https://docs.cloudbees.com domain. You should update any bookmarks for these URLs accordingly. (EC-13650)

  • The "Messages" tab in the Cluster Manager now displays more messages. The individual messages can be expanded to show more detail and collapsed. (EC-13761)

  • The --emake-crossmake option to emake is deprecated. (EC-13786)

  • Communication from emake to the Cluster Manager and agents to the Cluster Manager now uses SSL by default.

Resolved Issues

  • After sourcing ecloud.bash.profile, the RPM package manager doesn’t work because of the LD_LIBRARY_PATH settings. (EC-13751)

  • After installing the CloudBees Accelerator agent, the Gnome Terminal doesn’t work because of the LD_LIBRARY_PATH setting in /etc/profile/accelerator.sh. (kernel 4.18). (EC-13732)

  • Error message links now point to the CloudBees doc site. (EC-13745)

  • A deadlock in lofs_write_begin was fixed. (EC-13828)

  • Non-build processes are blocked from scanning lofs. (EC-13848)

  • (Linux platforms only) An issue where agent hosts crashed in lofs_threadfn is fixed. (EC-13757)

  • (Linux platforms only) An issue where agent hosts crashed in BufferFindRecord is fixed (EC-13749)

  • (Linux platforms only) An issue where agent hosts crashed in lofs_permission is fixed. (EC-13741)

  • (Linux platforms only) An issue where yum, rpm, and other system utilities failed after installation is fixed. (EC-13751)

  • (Windows platforms only) An incompatibility between electrify and 64-bit Cygwin processes is fixed. (EC-13714)

  • CloudBees Build Acceleration now has the ability to enable TLS communication between emake and the agents.

Other issues

  • (Windows platforms only) An issue where agents reported errors attemping to run Cygwin mount if another mount was earlier in PATH is fixed. (EC-13748)

  • (Red Hat Linux 7 platforms only) An issue that caused system crashes caused by LOFS on agent hosts is fixed. (EC-13741)

  • An eMake core dump that occurred when agents did not gracefully handle end-of-file symbols from eMake is now fixed. (EC-11599)

  • Emake core dump when running oracle precompiler is fixed. (EC-13708)

  • (Linux platforms only) An issue that caused system crashes on agent hosts in lofs_threadfn is fixed. (EC-13757)

  • (Linux platforms only) An issue that caused system crashes in BufferFindRecord on agent hosts is fixed. (EC-13749) *The emake to Cluster Manager and agent to Cluster Manager communication now uses TLS by default. (EC-13539)

  • CloudBees Build Acceleration is updated to use OpenSSL 1.1.1g on Linux. The Cluster Manager now uses Apache 2.4.43, PHP 7.4.8 and Jetty 9.4.30.v20200611. (EC-13837)

  • Supported Visual Studio version list updated in docs. (EC-13682)

  • An issue where Kernel modules would not build on Linux kernel 5.2 and up is fixed. (EC-13681)

  • Incorrect path to openssl and openssl.cnf in documentation has been fixed. (EC-13412)

  • Document build speedup in Cluster Manager. (EC-8084)

Platform support

Support is added for the following platforms:

  • RHEL 6.10, RHEL 7.8, 8.1, and 8.2

  • Ubuntu 19.04, 19.10, 20.04

  • CentOS 7.6, 7.6, 7.8, 8.0 and 8.1

  • Debian 10

The following platforms are no longer supported:

  • Linux 32-bit platforms

CloudBees Build Acceleration Insight for macOS is now available

CloudBees Build Acceleration Insight, the reporting and build visualization tool, is now available for the Mac on operating systems starting with Catalina.

Database support

No changes.

Browser support

No changes.

Installation and upgrade notes

Cloud bursting

Because the agent to Cluster Manager communication uses TLS by default, the agent image used for AWS EC2, GCP or Azure cloud bursting must also be upgraded to the 2020.07.00 release in order for cloud bursting to work.

New CloudBees Build Acceleration release strategy

Starting in August 2019, the release strategy for CloudBees Build Acceleration is updated to add a “preview” release in addition to the standard long-term support (LTS), maintenance (patch), and hotfix releases.

The release numbering for the preview releases uses a new <year>.<month>.00 numbering scheme. For example, 2019.10.00.

CloudBees recommends that you upgrade to the preview release and test these features in a controlled environment before rolling them out to production.

For details about the new release strategy, see the CloudBees Build Acceleration Release Strategy Update web page.

Version enforcement in Cluster Manager licensing

As of version 11.1, perpetual license files include a Version field and are therefore no longer transferable to succeeding product versions. At run time, to be considered valid by the Cluster Manager, the license file must have a Version that equals or exceeds the version of the Cluster Manager itself. Version checks consider only the major.minor version (for example, a license for 11.1 is valid for 11.1.0, 11.1.1, 11.1.2, and so on).

eMake authenticates with the Cluster Manager via the eMake/Cluster Manager protocol version to ensure that the eMake and Cluster Manager licenses match. If the license has no Version field, it will be considered invalid by the 11.1 Cluster Manager.

You can upgrade just the Cluster Manager, but starting with version 11.1, if you upgrade eMake or agents, you must also upgrade the Cluster Manager. When you upgrade the Cluster Manager, you must also acquire a new license if you are currently using a perpetual license.

You can upgrade to new patch releases (but not new feature releases) without acquiring a new license. For example, if your current license specifies 11.1, then you can use a Cluster Manager with version 11.1, 11.1.1, 11.1.2, and so on. Also, for example, if your license specifies 12.0, then you can use a Cluster Manager with version 12.0, 12.0.1, 12.0.2, and so on, as well as 11.x.

The Administration > Licenses page in the Cluster Manager web UI has a Version column, which provides the product version of each license in the list. This information is blank for a license with no explicit version (such as an evaluation license).

Make sure that you import a compliant license into the Cluster Manager before upgrading. For assistance, see https://support.cloudbees.com/. For details about importing a license, see Logging In and Enabling Licensing - Importing Your License. (EC-13170 and EC-13154)

Hardware requirements

  • The recommended total amount of RAM for an agent host is 2 GB per agent plus the amount of RAM normally needed to execute your build. For example, if you are running four agents, and your build normally needs 16 GB, you will need ((2 * 4) + 16) = 24 GB.

Backing up before you upgrade

  • The upgrade process does not preserve the existing files. Back up the /opt/ecloud/<arch>/cloud directory for Linux or the C:\ECloud\<arch> folder for Windows to a safe location.

  • For additional security, back up the database by following the recommended procedure from your database vendor.

Installing JDBC drivers for MySQL or Oracle Databases

CloudBees no longer distributes the JDBC drivers for MySQL or Oracle databases. To use one of these databases, you must download its driver directly from the Oracle website, then copy it to the appropriate directory on the Cluster Manager server, and then restart the Cluster Manager service. For details, see the Installing JDBC Drivers for MySQL or Oracle Databases section in the CloudBees Build Acceleration Installation Guide.

If you relocate eMake

If you copy the emake executable to a new location, you must also copy the execserver executable to that location. By default, the path to the execserver executable is /opt/ecloud/i686_Linux/32/bin/execserver (for 32-bit eMake) or /opt/ecloud/i686_Linux/64/bin/execserver (for 64-bit eMake).

Regenerating history files after an upgrade

The identifier that is used to find certain types of jobs in the eMake history file changed in version 8.0. After an upgrade from version 7.2.2 or older versions to version 8.0 or newer versions, users should regenerate their history files by running their first build with the --emake-history=create option to avoid unnecessary serializations. This build might have more conflicts than normal (but subsequent builds should return to normal).

Concurrent build licensing

As of version 9.1, for new CloudBees Build Acceleration subscription licenses, the number of builds that you can run concurrently is license-limited. The noLicenseWaitTime performance metric indicates the amount of time that a build spent waiting for a concurrent build license because the number of concurrent builds reached the license limit. Also, as of version 9.1, JobCache is not separately licensed and is now included with the concurrent build license.

Customers using pre-9.1 CloudBees Build Acceleration licenses may continue to use those licenses, including the licenses for the JobCache add-on.

For details about licensing for concurrent builds, see Logging In and Enabling Licensing. (EC-12095)

Known issues

Linux kernel issue that affects CloudBees Build Acceleration performance

Affected kernel versions

  • RHEL kernel versions later than 2.6.18-194.32 and earlier than 2.6.32-131

  • Ubuntu Linux kernel versions 2.6.31, 2.6.32, 2.6.33, and 2.6.34

Symptoms

Affected systems might encounter reduced performance on both ext3 and ext4 file systems. Symptoms might include

  • hung_task_timeout_secs messages in system dmesg logs

  • Widely variable agent availability (entering and exiting agent “penalty” status frequently)

  • Contention over the ecagent.state file

  • Slower builds (with unexplained variances)

To help determine if this issue exists, run the dmesg | grep hung_task_timeout command. hung_task_timeout errors show that this issue is present. Contact your kernel provider for another version of the precompiled kernel.

Fixes for systems running RHEL 5.6, 5.7, 5.8, and 6.0

You should consider upgrading to 2.6.32-131 (RHEL 6.1) or downgrading to 2.6.18-194.32 (RHEL 5.5).

Other known issues

  • Android M is not compatible with CloudBees Build Acceleration 11.1. You must use Android M with version 10.0.

  • If you kill a build manually, and the agents running on an Amazon EC2 instance fail to connect, the instance will continue to run. You must kill the instance manually.

  • If the Cluster Manager goes down while agents are running on an Amazon EC2 instance, they will still connect but will not be associated with a resource. You must kill the corresponding instance manually.

  • The cmtool importData command does not import license properties (such as maxAgents ). To work around this issue, re-import the license after using importData. (EC-12371)

  • You cannot control breakpoints from the Cluster Manager. (EC-12322)

  • If Apache fails to start properly after a new Cluster Manager installation, reboot the system.

  • An error occurs while attempting to upgrade cloudburst agent images to 11.3 when agent count is set to auto. (EC-13880)

Documentation

CloudBees Build Acceleration documentation is available on the CloudBees Build Acceleration product documentation page.

Cluster Manager online help and tooltips

Built into the Cluster Manager are

  • A complete, robust, context-sensitive online help system (click the Help button in any page of the Cluster Manager web UI). See the CloudBees Build Acceleration product documentation page for the latest updates to this information.

  • Tooltips with information to help fill in form fields.

Contacting CloudBees Support

To contact CloudBees Support, go to https://support.cloudbees.com/, and then click Submit a request to submit or see your support tickets.

Be prepared to provide your:

  • Name, title, company name, phone number, and email address

  • Operating system and version number

  • Product name and release version

  • Problem description

Ask CloudBees website

Go to https://ask.cloudbees.com/ — a member-moderated community forum where you can:

  • Ask and answer questions as well as comment on (and vote for) the questions of others and their answers

  • Get help with installation and configuration

  • Submit feedback

CloudBees Accelerator Knowledge Base

Go to CloudBees Accelerator Knowledge Base to find in-depth explanations of specific topics and solutions for specific problems.