KBEA-00171 - Using eMake to Accelerate BitBake Builds Using ElectricAccelerator 11.0.1

Article ID:360032820532
6 minute readKnowledge base

This KB article applies to ElectricAccelerator ("Accelerator") 11.0.1. Note that the functionality described here is deprecated in earlier versions of Accelerator.

For Accelerator version 11.1 and later, you should instead see the "BitBake" section in the "Third-Party Integrations" chapter of the CloudBees Accelerator Electric Make User Guide at https://docs.cloudbees.com/docs/cloudbees-build-acceleration/latest/emake-user-guide/.

Accelerator 11.0.1 provides the ebitbake command, which applies acceleration to BitBake builds such as Yocto Project builds. It provides caching for BitBake do_configure tasks and caching and build acceleration for BitBake do_compile tasks.

Prerequisites

Make sure that you have installed:

  • Accelerator 11.0.1

  • Yocto version between 2.2.4 (Morty) and 2.6.0 (Thud) inclusive

Also, Accelerator JobCache is recommended for the best performance. To use JobCache, you should have a JobCache license for as many concurrent builds as in your BitBake BB_NUMBER_THREADS setting (the maximum number of tasks that BitBake should run in parallel at a time). This allows all parts of the process to proceed without waiting for a JobCache license.

(Optional) Ensuring Consistent Environment Variables to Optimize the JobCache Hit Rate

To avoid excessive JobCache misses and slower successive builds, do not change your environment variables, except if an environment variable contains the path to the build directory or poky directory (then it will be "normalized"). This allows cache hits when you build in a different location from where the cache was generated.

After you source poky/oe-init-build-env, you must prepend the path to the ebitbake command to the PATH environment variable. This path is /opt/ecloud/i686_Linux/64/bin by default. This must be done the same way for every build to avoid an excessive JobCache miss rate.

Setting Up Your Accelerator Build Environment for BitBake

To set up your environment for BitBake builds:

  1. Source the oe-init-build-env script as normal.

    For example, enter:

     . poky/oe-init-build-env mybuilddir/

  2. Add the ebitbake path to the front of your PATH variable.

    For example, enter:

    export PATH=/opt/ecloud/i686_Linux/64/bin:$PATH

  3. Edit your BitBake conf/local.conf file and set the eMake environment variables.

    You must do this step only for your first build.

    ` +•Set +EMAKEFLAGS` to specify the Cluster Manager to use by entering:

    export EMAKEFLAGS+="--emake-cm=[:]"

    The host can be either an IP address or a machine name. For example, enter:

    export EMAKEFLAGS+="--emake-cm=develcluster3"



    Global EMAKEFLAGS must not include --emake-jobcache=gcc. This value is
    added automatically where appropriate for specific tasks. You have some
    control over this with configuration files.

    • (Optional) EMAKE_ROOT is set by default to include the build directory and the
    ` poky `directory, but you can change this variable in the file.

    • (Optional) EMAKE_ASSET_BASE is set by default to be emakeassets within the
    build directory.

    This variable points to the cache data, annotation files, and debug logs. You
    can change this variable to point to another location in the file. Note that
    the eMake asset directory might grow large with annotation files and cached
    build files.

Starting a BitBake Build

Run your build as normal but replace the bitbake command with ebitbake. The ebitbake command creates a build name such as BUILDNAME = 20190218125538. This is important for reporting, because it is used as a key to name annotation files. ebitbake displays the BUILDNAME value when it starts up.

(Optional) Selecting Packages or Tasks to Build With eMake

Accelerator support for BitBake builds provides control of build options not just for each package but also for each task of each package. You can control whether to use eMake for do_compile as well as do_configure tasks. For example, some tasks are too small to benefit from acceleration with eMake, so you should build them with gmake instead. You can specify these "gmake only" packages by creating a package_options file.

Also, for example, you can use a package_options file to use a specific emake option such as --emake-debug for a particular package during a particular task.

When a task is scheduled for execution, ebitbake uses the task name (such as do_compile), and then the package name to look up the set of options to apply. If the task is not listed, then it is treated in the same way that BitBake would treat it. If the package is not listed, then the default setting for that task is used.

A package_options file is a JSON file containing a dictionary at the top level of task types and a "default" setting to set the default behavior. This is a sample custom package_options file:

{
"do_compile" : {
    "default" : [ "emake", "#use emake unless told otherwise" ],
    "perl" : [ "gmake" ],
    "glibc" : [ "gmake" ],
},

"do_configure" : {
    "default" : [ "emake", "#use emake unless told otherwise" ],
    "qemu-native" : [ "emake", "subemake" ],
    "perl" : [ "gmake" ],
    "glibc" : [ "gmake" ],
    }
}

Example: To allowlist all do_compile tasks so that they are all compiled with eMake:

{
    "do_compile" : {
        "default" : [ "emake" ]
    }
}

Example: To denylist glibc so all other do_compile tasks use eMake:

{
    "do_compile" : {
        "default" : [ "emake" ],
        "glibc" : [ "gmake" ]
    }
}

Example: To allowlist python3-native so that only python3-native is built with eMake:

{
    "do_compile" : {
        "default" : [ "gmake" ],
        "python3-native" : [ "emake" ]
    }
}

Example: To cache the do_configure task of python3-native and build all code in the do_compile task with eMake:

{
    "do_configure" : {
        "default" : [ "gmake" ],
        "python3-native" : [ "emake" ]
    },

    "do_compile" : {
        "default" : [ "gmake" ],
        "python3-native" : [ "emake" ]
    }
}

Unsupported Yocto Project Modules

If no configuration is provided, ebitbake uses a reasonable set of defaults based on Electric Cloud test results and therefore builds certain packages normally with gmake. To see the default package build options, enter:

ebitbake --eb-showopt

Using Job Caching and "Learning" Builds when the Build Directory is New Each Time

Some continuous-integration systems create a new build directory for each build. You can use the cached files and eMake history from a previous build by setting the EMAKE_ASSET_BASE environment variable in your BitBake conf/local.conf file. You must do this before starting a learning build.

The steps are the same for starting a build, except that when editing conf/local.conf, you must also add a line similar to:

export EMAKE_ASSET_BASE="/path/to/some/common/location"

This location must be writable and must have enough space to store a large cache of the build outputs and the annotation files.

This functionality is not suitable for simultaneous access by more than one build.

Showing Which Package and Task Are Being Built in the Cluster Manager

You can show which package and task are being built in the Cluster Manager by adding the USER_BUILD_LABEL tag to the tag definition of the appropriate build class. The following example shows how to set the default build class:

default_%GC%_%DATE%_%USER_BUILD_LABEL%

where GC is a globally-unique number, DATE is the build start date and time, and USER_BUILD_LABEL is the label specified at the eMake command line (such as --emake-build-label=openssl-native_do_compile).

The following example shows how the resulting builds will appear in the Cluster Manager:

default_12_20190226045408_20190226125246_openssl-native_do_compile

For details about adding tag definitions to build classes, see the "Using Build Classes" section in the "Additional Electric Make Settings and Features" chapter of the ElectricAccelerator Electric Make User Guide at https://docs.cloudbees.com/docs/cloudbees-build-acceleration/latest/emake-user-guide/.

Viewing ebitbake Annotation, JobCache, History, and Log Files

These files are in the following locations.

File

Location

Annotation

$EMAKE_ASSET_BASE/package_name /platform /version /task_name /BUILD_NAME -NNN.anno where `task_name` is do_configure or do_compile.

Assets such as the job cache and scheduling data

$EMAKE_ASSET_BASE/package_name/platform/allversions/task_name/assets where `task_name` is do_configure or do_compile.

History

$EMAKE_ASSET_BASE/package_name/platform/allversions/task_name/emake.dat where `task_name` is do_configure or do_compile.

Debugging

tmp/work/platform/package_name/version/temp/log.task_name where `task_name` is do_configure or do_compile. This includes details of whether eMake was used for the specific task and what other package options were active for it. Lines starting with EA indicate ebitbake debug information. For example: EA build.exec_func_shell: package opts for 'do_compile_irda-utils'='['emake']'

Analyzing a Build Using ElectricInsight

You can use ElectricAccelerator ElectricInsight ("Insight") to visualize and perform certain types of analysis on builds that you executed with BitBake. This includes the ability to run certain reports to view BitBake dependency information and aggregate statistics.

Job Classification That Insight Uses for BitBake Task Types

The Insight job classifier includes first-class support for BitBake task types to allow it to identify BitBake tasks natively for conversion into eMake annotation. Insight recognizes several "classes" of "job" so that those "jobs" are identified correctly in the Insight UI.

Following is the list of BitBake task types that are identified by the job classifier (and thus recognized by Insight) and the classification that Insight uses for each:

BitBake Task Type Extension

ElectricInsight Job Classification

do_fetch do_unpack do_patch do_configure do_compile do_compile_kernelmodules do_install do_package do_package_qa do_package_write do_package_write_rpm do_packagedata do_populate_lic do_populate_sysroot do_build

fetch unpack patch configure compile compile install packaging testing packaging packaging packaging install install build

Creating a Single Annotation File from a BitBake Build to Facilitate Better Analysis with Insight

Each BitBake package and task that is accelerated by Accelerator creates an annotation file. You can combine these files and BitBake buildstats data into a single "pseudo-annotation" file to create an annotation view of the entire build by using the bb2anno utility. You can then load this file into Insight to run certain reports to see BitBake dependency information and to perform aggregate analysis.

For tasks that were not run using eMake, bb2anno creates a "job" representing the work and names it after the package and task that were involved, such as texinfo-dummy-native-1.0-r0.do_fetch. For tasks that were executed using eMake and also have annotation, the actual individual jobs are incorporated into the pseudo-annotation file.

bb2anno is in the /opt/ecloud/i686_Linux/unsupported/ directory by default. Following is the bb2anno command usage:

bb2anno -id  -annoroot  -buildstats  -buildlogs

The following example creates a compressed pseudo-annotation file:

~/bb2anno -id 20181212102014 -annoroot emakeassets -buildstats tmp/buildstats/20181212102014/ -buildlogs tmp/work | gzip > combined_20181212102014.anno.gz

Using Insight with a BitBake pseudo-annotation file requires Insight version 5.5 (which is bundled with Accelerator 11.0.1) and an Insight 5.5 license to generate reports from the file. For details about analyzing a build using Insight, see the CloudBees Build Acceleration Insight 5.5 User Guide at https://docs.cloudbees.com/docs/cloudbees-build-acceleration/latest/user-guide/.

Applies To

Accelerator 11.0.1.