ElectricAccelerator 11.0 Release Notes

10 minute read

Product Description

ElectricAccelerator® is a software build accelerator that dramatically reduces build times by distributing the build over a large cluster of inexpensive servers. ElectricAccelerator (“Accelerator”) uses a patented dependency-management system to identify and fix problems in real time that break traditional parallel builds. Accelerator plugs seamlessly into existing software development environments and includes web-based management and reporting tools.

Accelerator includes the following components:

  • Electric Make® (“eMake”)

  • Electric File System (EFS)

  • ElectricAccelerator Agents (“Agents”)

  • Cluster Manager

  • Electrify

What’s New or Modified

New Features and Functionality

Cloud Bursting

Support for public cloud-based resources is improved. This includes the addition of cloud bursting (combined static and dynamic resources) for Amazon EC2 instances and the introduction of Kubernetes support, which also includes a cloud bursting capability for containers. This release also moves the Cloud tab to the top level of the Cluster Manager web UI. (EC-12938 and EC-12913)

For details about these cloud bursting enhancements, see the ElectricAccelerator 11.0 Help for Cloud.

Cluster Manager Resource Enhancements

  • This release improves usability by simplifying, consolidating, and streamlining the entire resource concept:

    • Resources are now an integral part of the Cluster Manager—There is no “enabled” or “disabled;” they are simply “always on.”

    • There are only “built-in” resources, to use the terminology of the previous releases. The option to use an LSF resource manager or an EC2 resource manager, etc. is removed.

    • Resources are extended to optionally allow, on a per-resource basis, “cloud bursting” functionality when demand exceeds capacity.

      For details about these resource enhancements, see the _ElectricAccelerator 11.0 Cluster Manager Guide.

  • This release provides finer control over your cloud configuration by letting you perform configuration on a per-resource basis. (EC-12948)

  • You can now specify the number of agents per instance (except for Kubernetes, in which this value is always 1). (EC-13080)

  • You can now create extra (custom) tags to add to an Amazon EC2 instance via the Agents > Resources tab in the Cluster Manager web UI. For details, see the Resources - Create or Edit a Resource documentation. (EC-13021)

Cluster Manager Resource Statistics Enhancements

The tracking of resource statistics is improved to increase the accuracy of certain reports. Resource statistics tracking is updated as follows:

  • Time-weighted average and peak agent demand, agent shortage, and builds running is now tracked

  • Time-weighted average for agents available and agents in use is now tracked

  • Agent shortage is now computed based only on data points that have a non-zero agent shortage

  • Average build duration is no longer tracked

This change in the tracking of resource statistics affects the following reports:

  • Agent Usage—Now uses the peak agent demand and peak agents available

  • Cluster Utilization—Now uses the peak agent demand and peak running builds

  • Resource Usage over Time—Now uses the peak running builds per resource

  • Sea Level —Now uses the peak agent demand

For details about using these reports in the Cluster Manager, see the Reports topic in the documentation. (EC-12740)

Improved Support for Embedded Linux Builds

  • Accelerator and JobCache support of Yocto build acceleration is enhanced with a new ebitbake command. This functionality replaces GNU Make ( gmake ) with eMake do_compile build steps and runs gmake for other types of build steps. (EC-13026)

    The ebitbake command provides caching for BitBake do_configure tasks as well as caching and build acceleration for do_compile steps. The usage is almost exactly the same as BitBake itself. To improve Yocto compile JobCache hit rates, this command replaces the bemake wrapper utility, which was used in Accelerator 10.1.

    The functionality described above requires a JobCache license for the number of agents to be used in your builds and a Yocto version between 2.2.4 (latest current Morty) and 2.6.0 (Thud) inclusive.

  • The bb2anno utility is introduced. This utility converts the annotation files for each BitBake package and each task that uses Accelerator and combines it along with other buildstats data into a single annotation file. This lets you view the annotation in ElectricInsight (”Insight”) to show BitBake dependency information and aggregate statistics in certain reports.

    This functionality requires Insight version 5.5 (which is bundled with Accelerator) and an Insight 5.5 license to generate Insight reports from the annotation file. (EC-12908)

  • Build speed is improved by about 50 percent . (EC-xxx)

For more information about using eBitBake and the bb2anno utility, see the KBEA-00169 - Using eMake to Accelerate BitBake Builds Using ElectricAccelerator 11.0 and Newer Versions KB article. For details about analyzing a Yocto BitBake build using Insight, see the Electric Insight 5.5 User Guide.

Android Build Improvements

This release provides better performance and greater compatibility for Android builds as follows.

  • Common Android environment variables are added to the default parse avoidance “ignore” list when you use the the Android integration ( --emake-android-version); otherwise, you must specify them explicitly.

    For details, see Performance Optimization - Parse Avoidance. (EC-12953)

  • Android Open Source Project (AOSP) kernel builds are now faster because of the following improvements:

    • Support of patterns via wildcards in pragma files is added.

      For details, see Specifying Pragmas in an Addendum File. (EC-12905)

    • An issue where eMake Ninja emulation did not delete .d files (which slowed Android incremental builds) is fixed.

      eMake now automatically cleans up each target’s “depfile” as the build progresses, if you have specified both “depfile” and “deps” for the rule, and if you have enabled eDepend (autodep). This functionality applies only to cluster mode (not local mode). (EC-12910)

  • Ninja emulations now scan for hardcoded make or gmake (or, for example, /usr/bin/make ) in command lines used in Ninja builds and automatically replace them with emake --emake-emulation=gmake . Compound commands are now split at the && boundary to undo the bundling that Kati performed during translation from make to Ninja to avoid submake stub problems. (EC-12704)

  • Support for LineageOS is introduced.

Docker Support

Support for running Accelerator in Docker containers on Linux platforms is updated. For details, see Configuring ElectricAccelerator to Run in Docker Containers.

Visual Studio Integration Update

The Visual Studio integration (which is bundled with Accelerator) is upgraded to version 5.2.6. For the list of new features and resolved issues, see the “What’s New” section in the ElectricAccelerator 5.2.6 Visual Studio Integration Guide.

ElectricInsight Update

ElectricInsight (which is bundled with Accelerator) is upgraded to version 5.5. The 32- and 64-bit executables for ElectricInsight are installed.

ElectricInsight is licensed separately from Accelerator. An ElectricInsight license file is required to run reports. All other functionality (view-only mode) is available without a license.

For information about the 32- and 64-bit executables for ElectricInsight and about using ElectricInsight, see the ElectricInsight 5.5 User Guide.

Other New Features and Functionality

  • (Windows platforms only) Support for JobCache for CL in Electrify builds is introduced.

  • JobCache now uses an intelligent hash to ignore .NT_GNU_BUILD_ID differences in ELF executables and shared object libraries. (EC-13047 and EC-11228)

  • eDepend documentation should explain behavior when history file conflicts with pragma noautodep. (EC-13040)

  • Resources page in Accelerator Help docs missing purpose of resources. (EC-13005)

  • Common Jenkins/Hudson environment variables are added to the default parse avoidance “ignore” list. For details, see Electric Make User Guide - Performance Optimization - Parse Avoidance. (EC-12953) (EC-12952)

  • A concurrency license limit/count is added to the Concurrent Builds report. (EC-12921)

  • how emake-test-case-mode handles optimization data should be provided in the documentation. (EC-12870)

  • DOC CLONE - LINK : fatal error LNK1201: error writing to program database. (EC-12834)

  • (Linux platforms only) eMake now creates output files as sparse files if the underlying file system supports sparse files. (EC-12721)

  • Local-agent builds no longer produce excessive INFO-level messages. (EC-12696)

Behavior Changes

LSF and SGE

Support for Load Sharing Facility (LSF) and for Sun Grid Engine (SGE) is removed and replaced with the cloud bursting feature (described above). Following are the specific changes. (EC-13017 and EC-12937)

  • In the Configuring the Electric Agent screen of the GUI installer, the Agents are provisioned on-demand from a cloud or grid manager-Amazon EC2, LSF, etc. checkbox is changed to Agents are provisioned on-demand from a cloud - Amazon EC2 etc.

  • (Linux platforms only) In the interactive command-line installer, the Agents are provisioned on-demand from a cloud or grid manager-Amazon EC2, LSF, etc. [n/Y] prompt is changed to Agents are provisioned on-demand from a cloud - Amazon EC2 etc. [n/Y] .

  • In the silent installer, the --agentislsfhost option is replaced with --agentondemand .

  • For silent installer properties files, the ECLOUD_AGENT_IS_LSF_HOST variable is changed to EC_AGENT_IS_AGENT_ON_DEMAND .

  • In the ecconfig UI, the Is LSF Host text is replaced with Is On-demand Host .

Other Behavior Changes

  • In the Cluster Manager web UI, resource management is moved from the Administration > Server Settings tab to the Agents > Resources tab.

  • Priority pool functionality is removed. (EC-12937)

  • The Cluster Manager default for agent allocation policy is changed from “exclusive” to “shared.” (EC-12902)

Resolved Issues

Security-Related Issues

The following Cluster Manager web server security-related issues are fixed:

  • (Linux platforms only) PHP is upgraded from version 5.6.29 to version 5.6.37. (EC-12868)

  • JQuery is updated from version 1.11.1 to version 1.12.1. (EC-12840)

Other Issues

  • An issue in which eMake crashed because of incomplete dependency optimization is fixed. (EC-13125)

  • Reporting for errors during bootstrap setup when you use the --emake-android-version eMake option is improved. (EC-13063)

  • An issue where eMake produced a `ruleJob→getRunReason() == RuleJob::NOT_RUN' failed assertion and then aborted is fixed. (EC-13019)

  • MAKECMDGOALS is no longer missing from Ninja commands in the main_mk_overrides.mk file. (EC-13018)

  • An incompatibility with gmake 3.81 when handling ‘\’ in a makefile is fixed. (EC-12968)

  • The !job→getOrderString().empty() assertion failure is fixed. (EC-12922)

  • An issue that could cause persistent conflicts in jobs associated with intermediate targets is fixed. (EC-12961)

  • Updates for Android 9 patches are now included. (EC-12933)

  • An __assert_fail () error that caused eMake to crash is fixed. (EC-12928)

  • An issue where the Concurrent Builds report with a date range of two weeks or more produced an http 500 error when there were a lot of builds is fixed. (EC-12867)

  • An issue where low-priority builds using local agents sometimes failed with an EC1068: Unable to open file <path>: No such file or directory error is fixed. (EC-12863)

  • An issue in which agent workload data in emake_end messages exceeding 8 kilobytes, which caused Cluster Manager errors and timed-out builds, is fixed. (EC-12858)

  • Agents are now shut down cleanly, and the Electric File System (EFS) is unmounted with containers. (EC-12845)

  • Pagination now works properly on the Messages page when a filter is applied. (EC-12605)

  • The Cluster Manager no longer “leaks” concurrent build licenses when waiting builds are stopped via the web UI or cmtool. (EC-12912)

  • The performance of the getBuilds API for running builds is improved when the order , offset , or limit query modifiers are used. (EC-12886)

  • (Windows platforms only) A Windows kernel panic because of a null pointer dereference in LogToLocalSystem is fixed. (EC-12988)

  • An issue where a Ninja mkdir job error occurred when racing to create a directory with a job that created it as a symlink is fixed. (EC-12904)

  • A variation in the amount of non-cached output no longer triggers a JobCache assertion failure on cache hits. (EC-11475)

Platform Support

Support is added for the following platforms.

  • CentOS 7.4-1708 (kernel 3.10.0-693) and 7.5-1804 (kernel 3.10.0-862) (64-bit) (EC-12940)

  • Debian 9 (kernel 4.9) (64-bit) (EC-12942)

  • RHEL 6.10 (kernel 2.6.32-754) (64-bit for the Cluster Manager; 32- and 64-bit for eMake and agents) (EC-12989)

Installation and Upgrade Notes

Hardware Requirements

  • Accelerator 7.2 and newer versions require a Pentium 4 or newer processor when running in a 32-bit Solaris x 86 environment.

  • The recommended total amount of RAM for an agent host is 2 GB per agent plus the amount of RAM normally needed to execute your build. For example, if you are running four agents, and your build normally needs 16 GB, you will need ((2 * 4) + 16) = 24 GB.

Backing Up Before You Upgrade

  • The upgrade process does not preserve the existing files. Back up the /opt/ecloud/<arch>/cloud directory for Linux and Solaris or the C:\ECloud\<arch> folder for Windows to a safe location.

  • For additional security, back up the database by following the recommended procedure from your database vendor.

Installing JDBC Drivers for MySQL or Oracle Databases]

Electric Cloud no longer distributes the JDBC drivers for MySQL or Oracle databases. To use one of these databases, you must download its driver directly from the Oracle website, then copy it to the appropriate directory on the Cluster Manager server, and then restart the Cluster Manager service. For details, see Installing JDBC Drivers for MySQL or Oracle Databases.

Copying the execserver Executable to a New Location if You Relocate eMake

If you copy the emake executable to a new location, you must also copy the execserver executable to that location. By default, the path to the execserver executable is /opt/ecloud/i686_Linux/32/bin/execserver (for 32-bit eMake) or /opt/ecloud/i686_Linux/64/bin/execserver (for 64-bit eMake).

Regenerating History Files After an Upgrade

The identifier that is used to find certain types of jobs in the eMake history file changed in version 8.0. After an upgrade from version 7.2.2 or older versions to version 8.0 or newer versions, users should regenerate their history files by running their first build with the --emake-history=create option to avoid unnecessary serializations. This build might have more conflicts than normal (but subsequent builds should return to normal).

Concurrent Build Licensing

As of version 9.1, for new Accelerator subscription licenses, the number of builds that you can run concurrently is license-limited. The noLicenseWaitTime performance metric indicates the amount of time that a build spent waiting for a concurrent build license because the number of concurrent builds reached the license limit. Also, as of version 9.1, JobCache is not separately licensed and is now included with the concurrent build license.

Customers using pre-9.1 Accelerator licenses may continue to use those licenses, including the licenses for the JobCache add-on.

For details about licensing for concurrent builds, see Logging In and Enabling Licensing. (EC-12095)

Known Issues

Linux Kernel Issue That Affects Accelerator Performance

Affected Kernel Versions

  • RHEL kernel versions later than 2.6.18-194.32 and earlier than 2.6.32-131

  • Ubuntu Linux kernel versions 2.6.31, 2.6.32, 2.6.33, and 2.6.34

Symptoms

Affected systems might encounter reduced performance on both ext3 and ext4 file systems. Symptoms might include

  • hung_task_timeout_secs messages in system dmesg logs

  • Widely variable agent availability (entering and exiting agent “penalty” status frequently)

  • Contention over the ecagent.state file

  • Slower builds (with unexplained variances)

To help determine if this issue exists, run the dmesg | grep hung_task_timeout command. hung_task_timeout errors show that this issue is present. Contact your kernel provider for another version of the precompiled kernel.

Fixes for Systems Running RHEL 5.6, 5.7, 5.8, and 6.0

You should consider upgrading to 2.6.32-131 (RHEL 6.1) or downgrading to 2.6.18-194.32 (RHEL 5.5).

Other Known Issues

  • Android M is not compatible with Accelerator 11.0. You must use Android M with Accelerator 10.0.

  • If you kill a build manually, and the agents running on an Amazon EC2 instance fail to connect, the instance will continue to run. You must kill the instance manually.

  • If the Cluster Manager goes down while agents are running on an Amazon EC2 instance, they will still connect but will not be associated with a resource. You must kill the corresponding instance manually.

  • The cmtool importData command does not import license properties (such as maxAgents). To work around this issue, re-import the license after using importData. (EC-12371)

  • You cannot control breakpoints from the Cluster Manager. (EC-12322)

  • If Apache fails to start properly after a new Cluster Manager installation, reboot the system.

Documentation

Product Documentation

Accelerator documentation is available on the ElectricAccelerator 11.0 documentation site.

Documentation on the website is updated periodically.

Cluster Manager Online Help and Tooltips

Built into the Cluster Manager are

  • A complete, robust, context-sensitive online help system (click the Help button in any page of the Cluster Manager web UI). See the documentation website as described above for the latest updates to this information.

  • Tooltips with information to help fill in form fields.