CloudBees Accelerator 11.2 Release Notes

12 minute read

Product Description

CloudBees Accelerator is a software build accelerator that dramatically reduces build times by distributing the build over a large cluster of inexpensive servers. CloudBees Accelerator uses a patented dependency-management system to identify and fix problems in real time that break traditional parallel builds. CloudBees Accelerator plugs seamlessly into existing software development environments and includes web-based management and reporting tools.

CloudBees Accelerator includes the following components:

  • Electric Make (“eMake”)

  • Electric File System (EFS)

  • CloudBees Accelerator agents (“Agents”)

  • Cluster Manager

  • Electrify

What’s New or Modified

New Features and Functionality

Deployment on Google Cloud Platform Marketplace

You can now deploy CloudBees Accelerator on Google Cloud Platform (GCP) Marketplace. GCP Marketplace lets you start up software packages easily without having to manually configure the software, virtual machine instances, storage, or network settings.

For details about GCP Marketplace, see What is Google Cloud Platform Marketplace?. To get started, go to the GCP Marketplace page of the GCP Console and search for “CloudBees Accelerator” or go directly to the CloudBees Accelerator solution page at https://console.cloud.google.com/marketplace/details/cloudbees/cloudbees-accelerator.

Amazon EC2 Spot Instance Support for Agent Cloud Bursting

A Prefer Spot Instances checkbox is added to the New Resource page in the Resources tab of the Cluster Manager web UI. When this feature is enabled, Amazon EC2 Spot Instances will be launched for agent cloud bursting.

Spot Instances let you use spare Amazon EC2 computing capacity at a discount versus on-demand pricing. For more information about Spot Instances and how to enable them, see the Resources - Create or Edit a Resource topic in the Cluster Manager 11.2 Help.

Agent cloud bursting is supported only on Linux platforms.

Job Caching Improvements

  • JobCache now caches .so files (for jobs using GNU ld and GNU ld-compatible archivers). (EC-13535)

    For details, see the Job Caching section in the “Performance Optimization” chapter of the CloudBees Accelerator 11.2 Electric Make User Guide.

  • JobCache now caches .a and .la files for jobs using GNU ar and GNU ar-compatible archivers. (EC-13205)

  • For kati job caching on Android 9.x (”P”), the corresponding noautodep pragmas that enable autodependency prevention are now created (in /opt/ecloud/i686_Linux/conf/types/android/version/emake.pragmas) for common files for the ea_run_kati target. kati job caching was re-introduced in CloudBees Accelerator 11.1; this update also ensures that kati job caching works for Android 10. (EC-13522)

  • When you use --emake-jobcache= to enable JobCache (rather than using #pragma jobcache, eMake now ignores trailing digits and version numbers in filename extensions (such as 29 in framework.jar29 and .8.3.0 in libcryptopp.so.8.3.0) so that it can use those extensions when trying to intelligently determine the type of JobCache to apply to a job. (EC-13204)

Android Build Improvements

This release provides better performance and greater compatibility for Android builds as follows.

  • On Linux platforms, support for Android 10 (“Q”) builds is added. For Android first-time (“learning”) builds, eMake “kick starts” these builds by determining if you lack a history file and then providing a “kickstart” history file, which helps to reduce the build duration by mitigating the effect of a large number of conflicts. This release adds this capability as part of Android 10 support. (EC-13542)

  • The CloudBees Accelerator Android integration now supports Android builds that are invoked as components of another build. (EC-13402)

Installation and Configuration Improvements

This release contains several improvements to make it easier to install and configure CloudBees Accelerator for simple one-system installations for first-time or trial-basis users.

  • In the interactive command line and GUI installation modes, when you choose the Express installation type and select either a Full installation (which includes agents) or an Agent installation, the installer now displays how many agents that it will configure. (EC-13554)

  • When you install all CloudBees Accelerator components (eMake, agents, and the Cluster Manager) on the same machine in Express installer mode, the installer no longer prompts you for the hostname or IP address of your Cluster Manager. Instead, the installer automatically sets the Cluster Manager hostname to localhost. This means that you no longer must specify the --emake-cm=host[:port] eMake option when invoking a build on a system that is running all components (such as for testing or other nonproduction or trial-basis situations).

    The installer places the localhost hostname setting in the emake.conf configuration file for eMake to use. For upgrades, the installer moves your existing emake.conf configuration file to a backup file and creates a new emake.conf file to include this setting along with your other settings. (EC-13572 and EC-13552)

  • (Linux platforms only) Installation of prerequisite libraries is simplified: The installer now installs missing prerequisite libraries after prompting you to confirm. If installation of these libraries fails, the installer displays the commands that you enter to install them afterward.

    For a list of the prerequisite libraries for your Linux platform (as well as the commands to install them), see Supported Linux Platforms and Prerequisites. (EC-13550)

  • The CloudBees Accelerator installer now displays the command that you must run to add the location of the emake executable to your PATH environment variable. (EC-13533)

    • For Linux GUI installation mode, the installer’s final dialog box displays To add emake to the PATH - execute command source /opt/ecloud/i686_Linux/conf/ecloud.bash.profile. For Linux interactive command line installation mode, the installer displays this message as well.

    • For both Windows installation modes (GUI and silent), the installer automatically adds the full CloudBees Accelerator installation directory path to the PATH variable. Note that if you start the installer in silent mode from PowerShell Core or a cmd.exe command prompt, you must run a new independent session of pwsh.exe or cmd.exe (as applicable) to add the full CloudBees Accelerator installation directory path to the PATH variable.

  • (Linux platforms only) On systems that have an /etc/profile.d directory, the installer now creates an /etc/profile.d/accelerator.sh bash shell script, which automatically adds the CloudBees Accelerator /opt/ecloud/i686_Linux/64/bin directory to the PATH environment variable. This functionality makes it easier to invoke eMake and CloudBees Accelerator Insight because you do not need to enter the full paths to these executables. (EC-12098)

Improved Support for Embedded Linux Builds

  • Yocto 2.0 (“Jethro”), 2.1 (“Krogoth”), 2.7 (“Warrior”), and 3.0 (”Zeus”) are now supported. (EC-13426, EC-13632, and EC-13381)

  • Support for using the Link Time Optimization (LTO) build option with GCC is added. For more information about using CloudBees Accelerator for BitBake builds such as Yocto Project builds, see Integrating BitBake with CloudBees Accelerator. (EC-13333)

Support for JobCache in Electrify (Linux only)

(Linux platforms only) Support for JobCache in Electrify on Linux platforms is introduced. For details about this enhancement, see the Electrify chapter of the CloudBees Accelerator 11.2 Electric Make User Guide. (EC-13150)

ecmsbuild.exe Wrapper

The ecmsbuild.exe wrapper, which provides a simpler command syntax for using Electrify to build Visual Studio solutions and MSBuild projects from the command line, is introduced. For details, see Building Visual Studio Solutions from the Command Line - Using the ecmsbuild.exe Wrapper with Electrify from the Command Line” chapter of the CloudBees Accelerator Visual Studio Integration Guide.

Cluster Manager Web UI Reskin

The Cluster Manager web UI is reskinned to match that of other CloudBees product web interfaces. (EC-13393)

Build Progress Monitoring on the Console for Command-Line Builds

The `--emake-show-progress eMake option is introduced. This option outputs the build progress, the number of agents running jobs, and other statistics as well as error messages to the console. For details, see eMake Command-Line Options, Environment Variables, and Configuration File of the CloudBees Accelerator 11.2 Electric Make User Guide. This feature requires a VT100 or compatible terminal. (EC-13390)

Rebranding

Visual Studio Integration Update

The Visual Studio Integration (which is bundled with CloudBees Accelerator) is upgraded to version 5.4. For a list of new features and resolved issues in this release, see CloudBees Accelerator Cluster Manager Guide - Signing In To the Cluster Manager Web UI.

CloudBees Accelerator Insight Update

CloudBees Accelerator Insight, which is bundled with CloudBees Accelerator, is upgraded to version 5.6. This upgrade adds the ability to run the Root Conflicts report from the command line and adds support for annotation files that exceed 2 GB. The 32- and 64-bit executables for Insight are installed.

For details about this update, see the CloudBees Accelerator Insight 5.6 Release Notes. For information about the 32- and 64-bit executables for Insight and about using Insight, see the CloudBees Accelerator Insight 5.6 User Guide.

Other New Features and Functionality

  • eMake outside of a Kubernetes cluster can now use agent cloud bursting from that Kubernetes cluster. For details, see Configuring CloudBees Accelerator for Agent Cloud Bursting chapter of the CloudBees Accelerator 11.2 Configuration Guide. (EC-13639)

  • Performance of incremental builds is improved. (EC-13637)

  • The Build Wait Time report in the Cluster Manager web UI now includes the license wait time. (EC-13324)

  • 64-bit eMake and 64-bit Electrify no longer partition large annotation and debug log files into smaller file “chunks.” (EC-9529)

Behavior Changes

The Cloud Burst Status table in the Cluster Manager web UI is renamed to Current Instance Status, and its style is updated to match those of the Agents and Builds tables. (EC-13606)

Resolved Issues

Security-Related Issues

The following Cluster Manager web server security-related issues are fixed:

  • (Linux platforms only) Apache is upgraded to version 2.4.41. (EC-13670)

  • (Linux platforms only) OpenSSL is upgraded to version 1.1.1d. (EC-13670)

  • (Linux platforms only) PHP is upgraded to version 7.4.1. (EC-13671)

  • Persistent session cookies are now set over HTTPS, and passwords from form input fields are now sent over HTTPS. (EC-13448)

Other Issues

  • eMake Cygwin emulation now processes wildcard functions correctly when $(wildcard) is used with absolute Windows-style paths in Cygwin emulation mode. (EC-13615)

  • On Android 8.x, 9.x, and 10 builds, an issue is fixed in which $OUT_DIR/ was not used in the override rules in the patch to the build/core/main.mk file (but was hardcoded as out/ instead). (EC-13580)

  • On Android 10 builds, an issue is fixed in which the file name pattern filename.zip in the out/target/product/generic/obj/CONFIG/kati_packaging/dist.mk packaging dependencies file that CloudBees Accelerator generates should have been name.$USER.zip, where $USER is from BUILD_USER, which must be set in the environment when kati is executed. This issue triggered cache misses in the next eMake build. (EC-13577)

  • The “Configuring CloudBees Accelerator to Run on a Kubernetes Cluster” section is introduced. This section is in the “Configuring CloudBees Accelerator” chapter of the CloudBees Accelerator 11.2 Installation and Configuration Guide, available here. (EC-13560)

  • (Linux platforms only) An issue where Android builds failed with an unexpected EOF while looking for matching `)' error is fixed. (EC-13544)

  • (Windows platforms only) An error code caused by certain invalid characters in file names is fixed. (EC-13450)

  • In Ninja emulation mode, eMake now considers all outputs (not just the first output) of a job to determine if the job is up-to-date. (EC-13431)

  • (Linux platforms only) A NoSuchMethodError message that occurred when the first build with on-demand agents started on Google Cloud Platform (GCP) is fixed. (EC-13425)

  • An issue where builds hung when using agents on the Cluster Manager machine from a different eMake client is fixed. (EC-13296)

  • (SLES 15 platform only) The error, The upgrade of MariaDB has failed, that sometimes occurred at the end of Cluster Manager installation is fixed. (EC-13254)

  • The issue that caused the following error message is fixed: SessionEfsService::needFileData aborting session: couldn’t find host "": Success. (EC-13052)

Platform Support

Support is added for the following platforms.

  • RHEL 8.0 (kernel 4.18.0-80) (64-bit) (EC-13433)

  • RHEL 7.7 (kernel 3.10.0-1062) (64-bit) (EC-13658)

  • RHEL 7.6 (kernel 3.10.0-957) (64-bit) (EC-13658)

Support is ended for the following platforms.

  • 64-bit:

    • Windows 7

    • Windows Server 2008

    • RHEL 5.4

    • RHEL 7.1

    • SLES 11 SP2 (kernel 3.0.34) (64-bit)

    • SLES 11 SP1 (kernel 2.6.32) (64-bit)

    • SLES 11 (kernel 2.6.27) (64-bit)

    • Ubuntu Linux 14.10, 15.04, 15.10, 16.10, 17.04, and 17.10

  • All 32-bit (Linux and Windows)

  • All Cluster Manager installations on Windows

  • All Solaris

For more information about these support retirements, see CloudBees Accelerator Platform Support Retirements - 2019.

Database Support

No changes.

Browser Support

No changes.

Installation and Upgrade Notes

New CloudBees Accelerator Release Strategy

Starting in August 2019, the release strategy for CloudBees Accelerator is updated to add a “preview” release in addition to the standard long-term support (LTS), maintenance (patch), and hotfix releases.

The release numbering for the preview releases uses a new <year>.<month>.00 numbering scheme. For example, 2019.10.00.

CloudBees recommends that you upgrade to the preview release and test these features in a controlled environment before rolling them out to production.

For details about the new release strategy, see the CloudBees Accelerator Release Strategy Update web page.

Version Enforcement in Cluster Manager Licensing

As of version 11.1, perpetual license files include a Version field and are therefore no longer transferable to succeeding product versions. At run time, to be considered valid by the Cluster Manager, the license file must have a Version that equals or exceeds the version of the Cluster Manager itself. Version checks consider only the major.minor version (for example, a license for 11.1 is valid for 11.1.0, 11.1.1, 11.1.2, and so on).

eMake authenticates with the Cluster Manager via the eMake/Cluster Manager protocol version to ensure that the eMake and Cluster Manager licenses match. If the license has no Version field, it will be considered invalid by the 11.1 Cluster Manager.

You can upgrade just the Cluster Manager, but starting with version 11.1, if you upgrade eMake or agents, you must also upgrade the Cluster Manager. When you upgrade the Cluster Manager, you must also acquire a new license if you are currently using a perpetual license.

You can upgrade to new patch releases (but not new feature releases) without acquiring a new license. For example, if your current license specifies 11.1, then you can use a Cluster Manager with version 11.1, 11.1.1, 11.1.2, and so on. Also, for example, if your license specifies 12.0, then you can use a Cluster Manager with version 12.0, 12.0.1, 12.0.2, and so on, as well as 11.x.

The Administration > Licenses page in the Cluster Manager web UI has a Version column, which provides the product version of each license in the list. This information is blank for a license with no explicit version (such as an evaluation license).

Make sure that you import a compliant license into the Cluster Manager before upgrading. For assistance, see https://support.cloudbees.com/. For details about importing a license, see Logging In and Enabling Licensing - Importing Your License. (EC-13170 and EC-13154)

Hardware Requirements

The recommended total amount of RAM for an agent host is 2 GB per agent plus the amount of RAM normally needed to execute your build. For example, if you are running four agents, and your build normally needs 16 GB, you will need ((2 * 4) + 16) = 24 GB.

Backing Up Before You Upgrade

  • The upgrade process does not preserve the existing files. Back up the /opt/ecloud/<arch>/cloud directory for Linux or the C:\ECloud\<arch> folder for Windows to a safe location.

  • For additional security, back up the database by following the recommended procedure from your database vendor.

Installing JDBC Drivers for MySQL or Oracle Databases

CloudBees no longer distributes the JDBC drivers for MySQL or Oracle databases. To use one of these databases, you must download its driver directly from the Oracle website, then copy it to the appropriate directory on the Cluster Manager server, and then restart the Cluster Manager service. For details, see the Installing JDBC Drivers for MySQL or Oracle Databases section in the CloudBees Accelerator Installation Guide.

Copying the execserver Executable to a New Location if You Relocate eMake

If you copy the emake executable to a new location, you must also copy the execserver executable to that location. By default, the path to the execserver executable is /opt/ecloud/i686_Linux/64/bin/execserver.

Regenerating History Files After an Upgrade

The identifier that is used to find certain types of jobs in the eMake history file changed in version 8.0. After an upgrade from version 7.2.2 or older versions to version 8.0 or newer versions, users should regenerate their history files by running their first build with the --emake-history=create option to avoid unnecessary serializations. This build might have more conflicts than normal (but subsequent builds should return to normal).

Concurrent Build Licensing

As of version 9.1, for new CloudBees Accelerator subscription licenses, the number of builds that you can run concurrently is license-limited. The noLicenseWaitTime performance metric indicates the amount of time that a build spent waiting for a concurrent build license because the number of concurrent builds reached the license limit. Also, as of version 9.1, JobCache is not separately licensed and is now included with the concurrent build license.

Customers using pre-9.1 CloudBees Accelerator licenses may continue to use those licenses, including the licenses for the JobCache add-on.

For details about licensing for concurrent builds, see Logging In and Enabling Licensing. (EC-12095)

Known Issues

Linux Kernel Issue That Affects CloudBees Accelerator Performance

Affected Kernel Versions

This applies to RHEL kernel versions later than 2.6.18-194.32 and earlier than 2.6.32-131.

Symptoms

Affected systems might encounter reduced performance on both ext3 and ext4 file systems. Symptoms might include

  • hung_task_timeout_secs messages in system dmesg logs

  • Widely variable agent availability (entering and exiting agent “penalty” status frequently)

  • Contention over the ecagent.state file

  • Slower builds (with unexplained variances)

To help determine if this issue exists, run the dmesg | grep hung_task_timeout command. hung_task_timeout errors show that this issue is present. Contact your kernel provider for another version of the precompiled kernel.

Fixes for Systems Running RHEL 5.6, 5.7, 5.8, and 6.0

You should consider upgrading to 2.6.32-131 (RHEL 6.1) or downgrading to 2.6.18-194.32 (RHEL 5.5).

Other Known Issues

  • (Ubuntu 18.04 only) Installation of the Electric File System (EFS) driver from the Agent/EFS installer might fail with the following errors: Installing Linux LOFS…​ EFS installation error : Building new kernel module for 5.0.0-1025-gcp and Installing Linux LOFS…​ EFS installation error : Building new kernel module for 4.15.0-29-generic. (The full CloudBees Accelerator installer is unaffected.) (EC-13653)

  • If you kill a build manually, and the agents running on an Amazon EC2 instance fail to connect, the instance will continue to run. You must kill the instance manually.

  • If the Cluster Manager goes down while agents are running on an Amazon EC2 instance, they will still connect but will not be associated with a resource. You must kill the corresponding instance manually.

  • The cmtool importData command does not import license properties (such as maxAgents). To work around this issue, re-import the license after using importData. (EC-12371)

  • You cannot control breakpoints from the Cluster Manager. (EC-12322)

  • Apache might fail to start properly after a new Cluster Manager installation. To work around this issue, reboot the system.

Documentation

CloudBees Accelerator documentation is available on the CloudBees Accelerator product documentation page.

Cluster Manager Online Help and Tooltips

Built into the Cluster Manager are

  • A complete, robust, context-sensitive online help system (click the Help button in any page of the Cluster Manager web UI). See the CloudBees Accelerator product documentation page for the latest updates to this information.

  • Tooltips with information to help fill in form fields.