Integrating BitBake with CloudBees Build Acceleration

This chapter provides information about how CloudBees Build Acceleration integrates with BitBake.

CloudBees Build Acceleration provides the ebitbake command, which applies acceleration to BitBake builds such as Yocto Project builds. It provides caching for BitBake do_configure tasks and caching and build acceleration for BitBake do_compile tasks.

Prerequisites

Make sure that you have installed:

CloudBees Build Acceleration
A currently supported Yocto version. See CloudBees Build Acceleration supported platforms for details.

Also, CloudBees Build Acceleration JobCache is recommended for the best performance. To use JobCache, a JobCache license is required for as many concurrent builds as in your BitBake BB_NUMBER_THREADS setting (the maximum number of tasks that BitBake can run in parallel at a time). This allows all parts of the process to proceed without waiting for a JobCache license.

(Optional) Ensuring consistent environment variables to optimize the JobCache hit rate

To avoid excessive JobCache misses and slower successive builds, do not change your environment variables, except if an environment variable contains the path to the build directory or poky directory (then it will be “normalized”). This allows cache hits when you build in a different location from where the cache was generated.

After you source poky/oe-init-build-env, you must prepend the path to the ebitbake command to the PATH environment variable. This path is /opt/ecloud/i686_Linux/64/bin by default. This must be done the same way for every build to avoid an excessive JobCache miss rate.

Setting up your CloudBees Build Acceleration build environment for BitBake

To set up your environment for BitBake builds:

Source the oe-init-build-env script as normal.

For example, enter:
```
. poky/oe-init-build-env mybuilddir/▼
```
Add the ebitbake path to the front of your PATH variable.

For example, enter:
```
export PATH=/opt/ecloud/i686_Linux/64/bin:$PATH▼
```
Edit your BitBake conf/local.conf file to set the eMake environment variables.

You must do this step only for your first build. .. Set EMAKEFLAGS to specify the Cluster Manager to use by entering:

+
```
export EMAKEFLAGS+="--emake-cm=<host>[:<port>]"▼
```
+ where <host> can be either an IP address or a machine name. For example, enter:

+
```
export EMAKEFLAGS+="--emake-cm=develcluster3"▼
```
+ Global EMAKEFLAGS must not include --emake-jobcache=gcc. This value is added automatically where appropriate for specific tasks. You have some control over this with configuration files.
1. (Optional) EMAKE_ROOT is set by default to include the build directory and the poky directory, but you can change this variable in the file.
2. (Optional) EMAKE_ASSET_BASE is set by default to be emakeassets within the build directory.
  
  This variable points to the cache data, annotation files, and debug logs. You can change this variable to point to another location in the file. Note that the eMake asset directory might grow large with annotation files and cached build files.
  export PATH=/opt/ecloud/i686_Linux/64/bin:$PATH
  ▼

Starting a BitBake build

Run your build as normal but replace the bitbake command with ebitbake. The ebitbake command creates a build name such as BUILDNAME = 20190218125538. This is important for reporting, because it is used as a key to name annotation files. ebitbake displays the BUILDNAME value when it starts up. It is also created as an index directory under tmp/buildstats for each build.

(Optional) Selecting packages or tasks to build with eMake

CloudBees Build Acceleration support for BitBake builds provides control of build options not just for each package but also for each task of each package. You can control whether to use eMake for do_compile as well as do_configure tasks. For example, some tasks are too small to benefit from acceleration with eMake so build them with gmake instead. You can specify these gmake-only packages by creating a package_options file.

Also, for example, you can use a package_options file to use a specific emake option such as --emake-debug for a particular package during a particular task.

When a task is scheduled for execution, ebitbake uses the task name (such as do_compile ), and then the package name to look up the set of options to apply. If the task is not listed, then it is treated in the same way that BitBake would treat it. If the package is not listed, then the default setting for that task is used.

A package_options file is a JSON file containing a dictionary at the top level of task types and a "default" setting to set the default behavior. This is a sample custom package_options file:

{
  "do_compile": {
    "default": {"*": ["emake", "# use emake unless told otherwise"]},
    "perl": {"*": ["gmake"]},
    "glibc": {"*": ["gmake"]}
  },

  "do_configure": {
    "default": {"*": ["emake", "# use emake unless told otherwise"]},
    "qemu-native": {"*": ["emake", "subemake"]},
    "perl": {"*": ["gmake"]},
    "glibc": {"*": ["gmake"]}
  }
}▼

Example: To allowlist all `do_compile` tasks so that they are all compiled with eMake:

{
  "do_compile": {
    "default": {"*": ["emake"]}
  }
}▼

Example: To denylist `glibc` so all other `do_compile` tasks use eMake:

{
  "do_compile": {
    "default": {"*": ["emake"]},
    "glibc": {"*": ["gmake"]}
  }
}▼

Example: To allowlist `python3-native` so that only `python3-native` is built with eMake:

{
  "do_compile": {
    "default": {"*": ["gmake"]},
    "python3-native": {"*": ["emake"]}
  }
}▼

Example: To cache the `do_configure` task of `python3-native` and build all code in the `do_compile` task with eMake:

{
  "do_configure": {
    "default": {"*": ["gmake"]},
    "python3-native": {"*": ["emake"]}
  },

  "do_compile": {
    "default": {"*": ["gmake"]},
    "python3-native": {"*": ["emake"]}
  }
}▼

Unsupported Yocto project modules

If no configuration is provided, ebitbake uses a reasonable set of defaults based on CloudBees test results and therefore builds certain packages normally with gmake. To see the default package build options, enter:

ebitbake --eb-showopt▼

Using job caching and learning builds when the build directory is new each time

Some continuous-integration systems create a new build directory for each build. You can use the cached files and eMake history from a previous build by setting the EMAKE_ASSET_BASE environment variable in your BitBake conf/local.conf file. You must do this before starting a learning build.

The steps are the same for starting a build, except that when editing conf/local.conf, you must also add a line similar to:

export EMAKE_ASSET_BASE="/path/to/some/common/location"▼

This location must be writable and must have enough space to store a large cache of the build outputs and the annotation files.

This functionality is not suitable for simultaneous access by more than one build.

Showing which package and task are being built in the Cluster Manager

You can show which package and task are being built in the Cluster Manager by adding the USER_BUILD_LABEL tag to the tag definition of the appropriate build class. The following example shows how to set the default build class: default_%GC%%DATE%%USER_BUILD_LABEL%

where GC is a globally-unique number, DATE is the build start date and time, and USER_BUILD_LABEL is the label specified at the eMake command line (such as --emake-build-label=openssl-native_do_compile ).

The following example shows how the resulting builds will appear in the Cluster Manager:

default_12_20190226045408_20190226125246_openssl-native_do_compile▼

For details about adding tag definitions to build classes, see Using Build Classes.

Viewing ebitbake annotation, JobCache, history, and log files

These files are in the following locations.

File Location

File	Location
Annotation	`$EMAKE_ASSET_BASE/package_name/platform/version/task_name/BUILD_NAME-NNN.anno` where `task_name` is `do_configure` or `do_compile`.
Assets such as the job cache and scheduling data	`$EMAKE_ASSET_BASE/package_name/platform/allversions/task_name/assets` where `task_name` is `do_configure` or `do_compile`.
History	`$EMAKE_ASSET_BASE/package_name/platform/allversions/task_name/emake.dat` where `task_name` is `do_configure` or `do_compile`.
Debugging	`tmp/work/platform/package_name/version/temp/log.task_name` where `task_name` is `do_configure` or `do_compile`. This includes details of whether eMake was used for the specific task and what other package options were active for it. Lines starting with `EA` indicate `ebitbake` debug information. For example: `EA build.exec_func_shell: package opts for 'do_compile_irda-utils'='['emake']'`

Annotation

$EMAKE_ASSET_BASE/package_name/platform/version/task_name/BUILD_NAME-NNN.anno

where `task_name` is do_configure or do_compile.

Assets such as the job cache and scheduling data

$EMAKE_ASSET_BASE/package_name/platform/allversions/task_name/assets

where `task_name` is do_configure or do_compile.

History

$EMAKE_ASSET_BASE/package_name/platform/allversions/task_name/emake.dat

where `task_name` is do_configure or do_compile.

Debugging

tmp/work/platform/package_name/version/temp/log.task_name

where `task_name` is do_configure or do_compile. This includes details of whether eMake was used for the specific task and what other package options were active for it. Lines starting with EA indicate ebitbake debug information. For example:

EA build.exec_func_shell: package opts for 'do_compile_irda-utils'='['emake']'

Analyzing a build using Insight

You can use Insight to visualize and perform certain types of analysis on builds that you executed with BitBake. This includes the ability to run certain reports to view BitBake dependency information and aggregate statistics.

Job classification that Insight uses for BitBake task types

The Insight job classifier includes first-class support for BitBake task types to allow it to identify BitBake tasks natively for conversion into eMake annotation. Insight recognizes several “classes” of “job” so that those “jobs” are identified correctly in the Insight UI.

Following is the list of BitBake task types that are identified by the job classifier (and thus recognized by Insight) and the classification that Insight uses for each:

BitBake Task Type Extension	Insight Job Classification
do_fetch	fetch
do_unpack	unpack
do_patch	patch
do_configure	configure
do_compile	compile
do_compile_kernelmodules	compile
do_install	install
do_package	packaging
do_package_qa	testing
do_package_write	packaging
do_package_write_rpm	packaging
do_packagedata	packaging
do_populate_lic	install
do_populate_sysroot	install
do_build	build

BitBake Task Type Extension

Insight Job Classification

do_fetch

fetch

do_unpack

unpack

do_patch

patch

do_configure

configure

do_compile

compile

do_compile_kernelmodules

compile

do_install

install

do_package

packaging

do_package_qa

testing

do_package_write

packaging

do_package_write_rpm

packaging

do_packagedata

packaging

do_populate_lic

install

do_populate_sysroot

install

do_build

build

Creating a single annotation file from a BitBake build to facilitate better analysis with Insight

Each BitBake package and task that is accelerated by CloudBees Build Acceleration creates an annotation file. You can combine these files and BitBake buildstats data into a single ”pseudo-annotation” file to create an annotation view of the entire build by using the bb2anno utility. You can then load this file into Insight to run certain reports to see BitBake dependency information and to perform aggregate analysis.

For tasks that were not run using eMake, bb2anno creates a “job” representing the work and names it after the package and task that were involved, such as texinfo-dummy-native-1.0-r0.do_fetch. For tasks that were executed using eMake and also have annotation, the actual individual jobs are incorporated into the pseudo-annotation file.

bb2anno is in the /opt/ecloud/i686_Linux/unsupported/ directory by default. Following is the bb2anno command usage:

bb2anno -id <BUILDNAME> -annoroot <path_to_anno_files> -buildstats <path_to_buildstats> -buildlogs <path_to_build_logs>▼

The following example creates a compressed pseudo-annotation file:

~/bb2anno -id 20181212102014 -annoroot emakeassets -buildstats tmp/buildstats/20181212102014/ -buildlogs tmp/work | gzip > combined_20181212102014.anno.gz▼

Using Insight with a BitBake pseudo-annotation file requires Insight version 5.5 (which is bundled with CloudBees Build Acceleration) and an Insight 5.5 license to generate reports from the file. For details about analyzing a build using Insight, see Insight User Guide.

Integrating BitBake with CloudBees Build Acceleration

Prerequisites

(Optional) Ensuring consistent environment variables to optimize the JobCache hit rate

Setting up your CloudBees Build Acceleration build environment for BitBake

Starting a BitBake build

(Optional) Selecting packages or tasks to build with eMake

Example: To allowlist all do_compile tasks so that they are all compiled with eMake:

Example: To denylist glibc so all other do_compile tasks use eMake:

Example: To allowlist python3-native so that only python3-native is built with eMake:

Example: To cache the do_configure task of python3-native and build all code in the do_compile task with eMake:

Unsupported Yocto project modules

Using job caching and learning builds when the build directory is new each time

Showing which package and task are being built in the Cluster Manager

Viewing ebitbake annotation, JobCache, history, and log files

Analyzing a build using Insight

Job classification that Insight uses for BitBake task types

Creating a single annotation file from a BitBake build to facilitate better analysis with Insight

Example: To allowlist all `do_compile` tasks so that they are all compiled with eMake:

Example: To denylist `glibc` so all other `do_compile` tasks use eMake:

Example: To allowlist `python3-native` so that only `python3-native` is built with eMake:

Example: To cache the `do_configure` task of `python3-native` and build all code in the `do_compile` task with eMake: