2014-02-11 19:45:54 +00:00
|
|
|
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
|
|
|
|
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
|
|
|
|
|
|
|
|
<chapter id="user-manual-execution">
|
|
|
|
<title>Execution</title>
|
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
The primary purpose for running BitBake is to produce some kind
|
|
|
|
of output such as an image, a kernel, or a software development
|
|
|
|
kit.
|
2014-02-14 19:55:42 +00:00
|
|
|
Of course, you can execute the <filename>bitbake</filename>
|
|
|
|
command with options that cause it to execute single tasks,
|
|
|
|
compile single recipe files, capture or clear data, or simply
|
|
|
|
return information about the execution environment.
|
2014-02-11 19:45:54 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2014-02-14 19:55:42 +00:00
|
|
|
This chapter describes BitBake's execution process from start
|
|
|
|
to finish when you use it to create an image.
|
|
|
|
The execution process is launched using the following command
|
|
|
|
form:
|
|
|
|
<literallayout class='monospaced'>
|
|
|
|
$ bitbake <target>
|
|
|
|
</literallayout>
|
|
|
|
For information on the BitBake command and its options,
|
|
|
|
see the
|
|
|
|
"<link linkend='user-manual-command'>BitBake Command</link>
|
|
|
|
chapter.
|
2014-02-11 19:45:54 +00:00
|
|
|
</para>
|
|
|
|
|
2014-02-14 19:55:42 +00:00
|
|
|
<section id='parsing-the-base-configuration-metadata'>
|
|
|
|
<title>Parsing the Base Configuration Metadata</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The first thing BitBake does is parse base configuration
|
|
|
|
metadata.
|
|
|
|
Base configuration metadata consists of the
|
|
|
|
<filename>bblayers.conf</filename> file to determine what
|
|
|
|
layers BitBake needs to recognize, all necessary
|
|
|
|
<filename>layer.conf</filename> files (one from each layer),
|
|
|
|
and <filename>bitbake.conf</filename>.
|
2014-02-18 20:46:13 +00:00
|
|
|
The data itself is of various types:
|
|
|
|
<itemizedlist>
|
|
|
|
<listitem><para><emphasis>Recipes:</emphasis>
|
|
|
|
Details about particular pieces of software.
|
|
|
|
</para></listitem>
|
|
|
|
<listitem><para><emphasis>Class Data:</emphasis>
|
|
|
|
An abstraction of common build information
|
|
|
|
(e.g. how to build a Linux kernel).
|
|
|
|
</para></listitem>
|
|
|
|
<listitem><para><emphasis>Configuration Data:</emphasis>
|
|
|
|
Machine-specific settings, policy decisions,
|
|
|
|
and so forth.
|
|
|
|
Configuration data acts as the glue to bind everything
|
|
|
|
together.</para></listitem>
|
|
|
|
</itemizedlist>
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
The <filename>layer.conf</filename> files are used to
|
|
|
|
construct key variables such as
|
|
|
|
<link linkend='var-BBPATH'><filename>BBPATH</filename></link>
|
|
|
|
and
|
|
|
|
<link linkend='var-BBFILES'><filename>BBFILES</filename></link>.
|
|
|
|
<filename>BBPATH</filename> is used to search for
|
|
|
|
configuration and class files under
|
|
|
|
<filename>conf/</filename> and <filename>class/</filename>
|
|
|
|
directories, respectively.
|
|
|
|
<filename>BBFILES</filename> is used to find recipe files
|
|
|
|
(<filename>.bb</filename> and <filename>.bbappend</filename>).
|
|
|
|
If there is no <filename>bblayers.conf</filename> file,
|
|
|
|
it is assumed the user has set the <filename>BBPATH</filename>
|
|
|
|
and <filename>BBFILES</filename> directly in the environment.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Next, the <filename>bitbake.conf</filename> file is searched
|
|
|
|
using the <filename>BBPATH</filename> variable that was
|
|
|
|
just constructed.
|
|
|
|
The <filename>bitbake.conf</filename> file usually indicates
|
|
|
|
all the other key include files to parse.
|
|
|
|
The usual convention is to have machine, distro, site, and local
|
|
|
|
configurations.
|
|
|
|
This means a user provides their own customizations
|
|
|
|
through a <filename>local.conf</filename> file.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
As mentioned in the previous paragraph, two of the other notable
|
|
|
|
configuration files are the distro and machine configuration
|
|
|
|
files.
|
2014-02-14 19:55:42 +00:00
|
|
|
These configuration files are normally identified by
|
|
|
|
variables unique to the build systems using BitBake.
|
|
|
|
For example, the Yocto Project uses the
|
|
|
|
<filename>DISTRO</filename> and <filename>MACHINE</filename>
|
|
|
|
variables, respectively.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Prior to parsing configuration files, Bitbake looks
|
|
|
|
at certain variables, including:
|
|
|
|
<itemizedlist>
|
|
|
|
<listitem><para><link linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link></para></listitem>
|
|
|
|
<listitem><para><link linkend='var-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link></para></listitem>
|
|
|
|
<listitem><para><link linkend='var-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link></para></listitem>
|
|
|
|
<listitem><para><link linkend='var-BB_ORIGENV'><filename>BB_ORIGENV</filename></link></para></listitem>
|
|
|
|
<listitem><para><link linkend='var-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></link></para></listitem>
|
|
|
|
<listitem><para><link linkend='var-PREFERRED_PROVIDERS'><filename>PREFERRED_PROVIDERS</filename></link></para></listitem>
|
|
|
|
</itemizedlist>
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The base configuration metadata is global
|
|
|
|
and therefore affects all packages and tasks that are executed.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
BitBake first searches the current working directory for an
|
|
|
|
optional <filename>conf/bblayers.conf</filename> configuration file.
|
|
|
|
This file is expected to contain a
|
|
|
|
<link linkend='var-BBLAYERS'><filename>BBLAYERS</filename></link>
|
|
|
|
variable that is a space delimited list of 'layer' directories.
|
2014-02-18 20:46:13 +00:00
|
|
|
Recall that if BitBake cannot find a <filename>bblayers.conf</filename>
|
|
|
|
file then it is assumed the user has set the <filename>BBPATH</filename>
|
|
|
|
and <filename>BBFILES</filename> directly in the environment.
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
For each directory (layer) in this list, a <filename>conf/layer.conf</filename>
|
|
|
|
file is searched for and parsed with the
|
|
|
|
<link linkend='var-LAYERDIR'><filename>LAYERDIR</filename></link>
|
|
|
|
variable being set to the directory where the layer was found.
|
|
|
|
The idea is these files automatically setup
|
|
|
|
<link linkend='var-BBPATH'><filename>BBPATH</filename></link>
|
|
|
|
and other variables correctly for a given build directory.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
BitBake then expects to find the <filename>conf/bitbake.conf</filename>
|
|
|
|
file somewhere in the user-specified <filename>BBPATH</filename>.
|
|
|
|
That configuration file generally has include directives to pull
|
|
|
|
in any other metadata such as files specific to the architecture,
|
|
|
|
the machine, the local environment, and so forth.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Only variable definitions and include directives are allowed
|
|
|
|
in <filename>.conf</filename> files.
|
|
|
|
The following variables include:
|
|
|
|
<itemizedlist>
|
|
|
|
<listitem><para>
|
|
|
|
<link linkend='var-BITBAKE_UI'><filename>BITBAKE_UI</filename></link>
|
|
|
|
</para></listitem>
|
|
|
|
<listitem><para>
|
|
|
|
<link linkend='var-BBDEBUG'><filename>BBDEBUG</filename></link>
|
|
|
|
</para></listitem>
|
|
|
|
<listitem><para>
|
|
|
|
<link linkend='var-MULTI_PROVIDER_WHITELIST'><filename>MULTI_PROVIDER_WHITELIST</filename></link>
|
|
|
|
</para></listitem>
|
|
|
|
<listitem><para>
|
|
|
|
<link linkend='var-BB_NUMBER_PARSE_THREADS'><filename>BB_NUMBER_PARSE_THREADS</filename></link>
|
|
|
|
</para></listitem>
|
|
|
|
<listitem><para>
|
|
|
|
<filename>BBPKGS</filename>
|
|
|
|
</para></listitem>
|
|
|
|
<listitem><para>
|
|
|
|
<link linkend='var-BB_DEFAULT_TASK'><filename>BB_DEFAULT_TASK</filename></link>
|
|
|
|
</para></listitem>
|
|
|
|
<listitem><para>
|
|
|
|
<link linkend='var-TOPDIR'><filename>TOPDIR</filename></link>
|
|
|
|
</para></listitem>
|
|
|
|
<listitem><para>
|
|
|
|
<link linkend='var-BB_VERBOSE_LOGS'><filename>BB_VERBOSE_LOGS</filename></link>
|
|
|
|
</para></listitem>
|
|
|
|
<listitem><para>
|
|
|
|
<link linkend='var-BB_NICE_LEVEL'><filename>BB_NICE_LEVEL</filename></link>
|
|
|
|
</para></listitem>
|
|
|
|
<listitem><para>
|
|
|
|
<link linkend='var-BBFILE_COLLECTIONS'><filename>BBFILE_COLLECTIONS</filename></link>
|
|
|
|
</para></listitem>
|
|
|
|
<listitem><para>
|
|
|
|
<link linkend='var-ASSUME_PROVIDED'><filename>ASSUME_PROVIDED</filename></link>
|
|
|
|
</para></listitem>
|
|
|
|
<listitem><para>
|
|
|
|
<link linkend='var-BB_DANGLINGAPPENDS_WARNONLY'><filename>BB_DANGLINGAPPENDS_WARNONLY</filename></link>
|
|
|
|
</para></listitem>
|
|
|
|
<listitem><para>
|
|
|
|
<link linkend='var-BBINCLUDED'><filename>BBINCLUDED</filename></link>
|
|
|
|
</para></listitem>
|
|
|
|
<listitem><para>
|
|
|
|
<link linkend='var-BBFILE_PRIORITY'><filename>BBFILE_PRIORITY</filename></link>
|
|
|
|
</para></listitem>
|
|
|
|
<listitem><para>
|
|
|
|
<link linkend='var-BUILDNAME'><filename>BUILDNAME</filename></link>
|
|
|
|
</para></listitem>
|
|
|
|
<listitem><para>
|
|
|
|
<link linkend='var-BBMASK'><filename>BBMASK</filename></link>
|
|
|
|
</para></listitem>
|
|
|
|
</itemizedlist>
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
After parsing configuration files, BitBake uses its rudimentary
|
|
|
|
inheritance mechanism, which is through class files, to inherit
|
|
|
|
some standard classes.
|
|
|
|
BitBake parses a class when the inherit directive responsible
|
|
|
|
for getting that class is encountered.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The <filename>base.bbclass</filename> file is always included.
|
|
|
|
Other classes that are specified in the configuration using the
|
|
|
|
<link linkend='var-INHERIT'><filename>INHERIT</filename></link>
|
|
|
|
variable are also included.
|
|
|
|
BitBake searches for class files in a "classes" subdirectory under
|
|
|
|
the paths in <filename>BBPATH</filename> in the same way as
|
|
|
|
configuration files.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
A good way to get an idea of the configuration files and
|
|
|
|
the class files used in your execution environment is to
|
|
|
|
run the following BitBake command:
|
|
|
|
<literallayout class='monospaced'>
|
|
|
|
$ bitbake -e > mybb.log
|
|
|
|
</literallayout>
|
|
|
|
Examining the top of the <filename>mybb.log</filename>
|
|
|
|
shows you the many configuration files and class files
|
|
|
|
used in your execution environment.
|
|
|
|
</para>
|
|
|
|
</section>
|
|
|
|
|
2014-02-18 20:46:13 +00:00
|
|
|
<section id='locating-and-parsing-recipes'>
|
|
|
|
<title>Locating and Parsing Recipes</title>
|
2014-02-14 19:55:42 +00:00
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
During the configuration phase, BitBake will have
|
|
|
|
set
|
|
|
|
<link linkend='var-BBFILES'><filename>BBFILES</filename></link>.
|
|
|
|
BitBake now uses it to construct a list of recipes to parse,
|
|
|
|
along with any append files (<filename>.bbappend</filename>)
|
|
|
|
to apply.
|
|
|
|
<filename>BBFILES</filename> is a space-separated list of
|
|
|
|
available files and supports wildcards.
|
|
|
|
An example would be:
|
2014-02-14 19:55:42 +00:00
|
|
|
<literallayout class='monospaced'>
|
|
|
|
BBFILES = "/path/to/bbfiles/*.bb"
|
|
|
|
</literallayout>
|
2014-02-18 20:46:13 +00:00
|
|
|
BitBake parses each recipe and append file located
|
|
|
|
with <filename>BBFILES</filename> and stores the values of
|
|
|
|
various variables into the datastore.
|
|
|
|
For each file, a fresh copy of the base configuration is
|
|
|
|
made, then the recipe is parsed line by line.
|
|
|
|
Any inherit statements cause BitBake to find and
|
|
|
|
then parse class files (<filename>.bbclass</filename>)
|
|
|
|
using
|
|
|
|
<link linkend='var-BBPATH'><filename>BBPATH</filename></link>
|
|
|
|
as the search path.
|
|
|
|
Finally, BitBake parses in order any append files found in
|
|
|
|
<filename>BBFILES</filename>.
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
One common convention is to use the recipe filename to define
|
|
|
|
pieces of metadata.
|
|
|
|
For example, in <filename>bitbake.conf</filename> the recipe
|
|
|
|
name and version set
|
|
|
|
<link linkend='var-PN'><filename>PN</filename></link> and
|
|
|
|
<link linkend='var-PV'><filename>PV</filename></link>:
|
|
|
|
<literallayout class='monospaced'>
|
|
|
|
PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE'),d)[1] or '1.0'}"
|
|
|
|
PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE'),d)[0] or 'defaultpkgname'}"
|
|
|
|
</literallayout>
|
|
|
|
In this example, a recipe called "something_1.2.3.bb" sets
|
|
|
|
<filename>PN</filename> to "something" and
|
|
|
|
<filename>PV</filename> to "1.2.3".
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
By the time parsing is complete for a recipe, BitBake
|
|
|
|
has a list of tasks that the recipe defines and a set of
|
|
|
|
data consisting of keys and values.
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
BitBake does not need all this information.
|
|
|
|
It only needs a small subset of the information to make
|
|
|
|
decisions about the recipe.
|
|
|
|
Consequently, BitBake caches the values in which it is
|
|
|
|
interested.
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
Subsequent BitBake commands then parse the base
|
|
|
|
configuration and compute a checksum of that data.
|
|
|
|
If that checksum matches what is in the cache, the
|
|
|
|
recipe and class files have not changed.
|
|
|
|
In this case, BitBake reloads the cached information
|
|
|
|
about the recipe instead of reparsing it from scratch.
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
|
|
|
</section>
|
|
|
|
|
2014-02-18 20:46:13 +00:00
|
|
|
<section id='bb-bitbake-providers'>
|
|
|
|
<title>Preferences and Providers</title>
|
2014-02-14 19:55:42 +00:00
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
Assuming BitBake has been instructed to execute a target and
|
|
|
|
that all the recipe files have been parsed, BitBake starts to
|
|
|
|
build the target and look for providers of that target.
|
|
|
|
Once a provider is selected, BitBake resolves all the dependencies for
|
|
|
|
the target.
|
|
|
|
As an example, suppose the target is
|
|
|
|
<filename>core-image-sato</filename>.
|
|
|
|
In this case, it would lead to
|
|
|
|
<filename>packagegroup-core-x11-sato</filename>,
|
|
|
|
which in turn leads to recipes like <filename>matchbox-terminal</filename>,
|
|
|
|
<filename>pcmanfm</filename> and <filename>gthumb</filename>.
|
|
|
|
These recipes in turn depend on <filename>eglibc</filename> and the toolchain.
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
Sometimes a target might have multiple providers.
|
|
|
|
A common example is "virtual/kernel", which is provided by each kernel package.
|
|
|
|
Each machine often selects the best kernel provider by using a line similar to the
|
|
|
|
following in the machine configuration file:
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
|
|
|
|
2014-02-18 20:46:13 +00:00
|
|
|
<literallayout class='monospaced'>
|
|
|
|
PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
|
|
|
|
</literallayout>
|
|
|
|
|
2014-02-14 19:55:42 +00:00
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
The default
|
|
|
|
<link linkend='var-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>
|
|
|
|
is the provider with the same name as the target.
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
|
|
|
|
2014-02-18 20:46:13 +00:00
|
|
|
<para>
|
|
|
|
Understanding how providers are chosen is made complicated by the fact
|
|
|
|
that multiple versions might exist.
|
|
|
|
BitBake defaults to the highest version of a provider.
|
|
|
|
Version comparisons are made using the same method as Debian.
|
|
|
|
You can use the
|
|
|
|
<link linkend='var-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></link>
|
|
|
|
variable to specify a particular version (usually in the distro configuration).
|
|
|
|
You can influence the order by using the
|
|
|
|
<link linkend='var-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
|
|
|
|
variable.
|
|
|
|
By default, files have a preference of "0".
|
|
|
|
Setting the <filename>DEFAULT_PREFERENCE</filename> to "-1" makes the
|
|
|
|
package unlikely to be used unless it is explicitly referenced.
|
|
|
|
Setting the <filename>DEFAULT_PREFERENCE</filename> to "1" makes it likely the package is used.
|
|
|
|
<filename>PREFERRED_VERSION</filename> overrides any <filename>DEFAULT_PREFERENCE</filename> setting.
|
|
|
|
<filename>DEFAULT_PREFERENCE</filename> is often used to mark newer and more experimental package
|
|
|
|
versions until they have undergone sufficient testing to be considered stable.
|
|
|
|
</para>
|
2014-02-14 19:55:42 +00:00
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
In summary, BitBake has created a list of providers, which is prioritized, for each target.
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
2014-02-18 20:46:13 +00:00
|
|
|
</section>
|
|
|
|
|
|
|
|
<section id='bb-bitbake-dependencies'>
|
|
|
|
<title>Dependencies</title>
|
2014-02-14 19:55:42 +00:00
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
Each target BitBake builds consists of multiple tasks such as
|
|
|
|
<filename>fetch</filename>, <filename>unpack</filename>,
|
|
|
|
<filename>patch</filename>, <filename>configure</filename>,
|
|
|
|
and <filename>compile</filename>.
|
|
|
|
For best performance on multi-core systems, BitBake considers each
|
|
|
|
task as an independent
|
|
|
|
entity with its own set of dependencies.
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
Dependencies are defined through several variables.
|
|
|
|
You can find information about variables BitBake uses in
|
|
|
|
the <link linkend='ref-variables-glos'>Variables Glossary</link>
|
|
|
|
near the end of this manual.
|
|
|
|
At a basic level, it is sufficient to know that BitBake uses the
|
|
|
|
<link linkend='var-DEPENDS'><filename>DEPENDS</filename></link> and
|
|
|
|
<link linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link> variables when
|
|
|
|
calculating dependencies.
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
For more information on how BitBake handles dependencies, see the
|
|
|
|
"<link linkend='dependencies'>Dependencies</link>" section.
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
|
|
|
</section>
|
|
|
|
|
2014-02-18 20:46:13 +00:00
|
|
|
<section id='ref-bitbake-tasklist'>
|
|
|
|
<title>The Task List</title>
|
2014-02-14 19:55:42 +00:00
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
Based on the generated list of providers and the dependency information,
|
|
|
|
BitBake can now calculate exactly what tasks it needs to run and in what
|
|
|
|
order it needs to run them.
|
|
|
|
The build now starts with BitBake forking off threads up to the limit set in the
|
|
|
|
<link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
|
|
|
|
variable.
|
|
|
|
BitBake continues to fork threads as long as there are tasks ready to run,
|
|
|
|
those tasks have all their dependencies met, and the thread threshold has not been
|
|
|
|
exceeded.
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
It is worth noting that you can greatly speed up the build time by properly setting
|
|
|
|
the <filename>BB_NUMBER_THREADS</filename> variable.
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
2014-02-11 19:45:54 +00:00
|
|
|
|
2014-02-14 19:55:42 +00:00
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
As each task completes, a timestamp is written to the directory specified by the
|
|
|
|
<link linkend='var-STAMP'><filename>STAMP</filename></link> variable.
|
|
|
|
On subsequent runs, BitBake looks in the build directory within
|
|
|
|
<filename>tmp/stamps</filename>and does not rerun
|
|
|
|
tasks that are already completed unless a timestamp is found to be invalid.
|
|
|
|
Currently, invalid timestamps are only considered on a per
|
|
|
|
recipe file basis.
|
|
|
|
So, for example, if the configure stamp has a timestamp greater than the
|
|
|
|
compile timestamp for a given target, then the compile task would rerun.
|
|
|
|
Running the compile task again, however, has no effect on other providers
|
|
|
|
that depend on that target.
|
|
|
|
This behavior could change or become configurable in future versions of BitBake.
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
|
|
|
|
2014-02-18 20:46:13 +00:00
|
|
|
<note>
|
|
|
|
Some tasks are marked as "nostamp" tasks.
|
|
|
|
No timestamp file is created when these tasks are run.
|
|
|
|
Consequently, "nostamp" tasks are always rerun.
|
|
|
|
</note>
|
|
|
|
|
2014-02-14 19:55:42 +00:00
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
For more information on tasks, see the
|
|
|
|
"<link linkend='tasks'>Tasks</link>" section.
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
2014-02-11 19:45:54 +00:00
|
|
|
</section>
|
|
|
|
|
2014-02-18 20:46:13 +00:00
|
|
|
<section id='executing-tasks'>
|
|
|
|
<title>Executing Tasks</title>
|
2014-02-14 19:55:42 +00:00
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
Tasks can either be a shell task or a Python task.
|
|
|
|
For shell tasks, BitBake writes a shell script to
|
|
|
|
<filename>${WORKDIR}/temp/run.do_taskname.pid</filename>
|
|
|
|
and then executes the script.
|
|
|
|
The generated shell script contains all the exported variables,
|
|
|
|
and the shell functions with all variables expanded.
|
|
|
|
Output from the shell script goes to the file
|
|
|
|
<filename>${WORKDIR}/temp/log.do_taskname.pid</filename>.
|
|
|
|
Looking at the expanded shell functions in the run file and
|
|
|
|
the output in the log files is a useful debugging technique.
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
For Python tasks, BitBake executes the task internally and logs
|
|
|
|
information to the controlling terminal.
|
|
|
|
Future versions of BitBake will write the functions to files
|
|
|
|
similar to the way shell tasks are handled.
|
|
|
|
Logging will be handled in a way similar to shell tasks as well.
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
Variables specific to scheduling functionality exist:
|
2014-02-14 19:55:42 +00:00
|
|
|
<itemizedlist>
|
2014-02-18 20:46:13 +00:00
|
|
|
<listitem><para>
|
|
|
|
<link linkend='var-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
|
2014-02-14 19:55:42 +00:00
|
|
|
</para></listitem>
|
2014-02-18 20:46:13 +00:00
|
|
|
<listitem><para>
|
|
|
|
<link linkend='var-BB_SCHEDULERS'><filename>BB_SCHEDULERS</filename></link>
|
2014-02-14 19:55:42 +00:00
|
|
|
</para></listitem>
|
|
|
|
</itemizedlist>
|
|
|
|
</para>
|
|
|
|
</section>
|
|
|
|
|
2014-02-18 20:46:13 +00:00
|
|
|
<section id='setscene'>
|
|
|
|
<title>Setscene</title>
|
2014-02-14 19:55:42 +00:00
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
This section needs to get the concept of the setscene across.
|
|
|
|
The reader needs to know what it is and what it is used for during
|
|
|
|
the build process.
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
You can find more information on setscene metadata in the
|
|
|
|
"<link linkend='task-checksums-and-setscene'>Task Checksums and Setscene</link>"
|
|
|
|
section.
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
2014-02-18 20:46:13 +00:00
|
|
|
</section>
|
|
|
|
|
|
|
|
<section id='checksums'>
|
|
|
|
<title>Checksums (Signatures)</title>
|
2014-02-14 19:55:42 +00:00
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
A checksum is a unique signature of a task's inputs.
|
|
|
|
The setscene code uses a checksum to determine if a task needs
|
|
|
|
to be run.
|
|
|
|
Because it is a change in a task's inputs that triggers running
|
|
|
|
the task, the process needs to detect all the inputs to a given task.
|
|
|
|
For shell tasks, this turns out to be fairly easy because
|
|
|
|
BitBake generates a "run" shell script for each task and
|
|
|
|
it is possible to create a checksum that gives you a good idea of when
|
|
|
|
the task's data changes.
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
To complicate the problem, some things should not be included in
|
|
|
|
the checksum.
|
|
|
|
First, there is the actual specific build path of a given task -
|
|
|
|
the working directory.
|
|
|
|
It does not matter if the working directory changes because it should not
|
|
|
|
affect the output for target packages.
|
|
|
|
The simplistic approach for excluding the working directory is to set
|
|
|
|
it to some fixed value and create the checksum for the "run" script.
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
Another problem results from the "run" scripts containing functions that
|
|
|
|
might or might not get called.
|
|
|
|
The incremental build solution contains code that figures out dependencies
|
|
|
|
between shell functions.
|
|
|
|
This code is used to prune the "run" scripts down to the minimum set,
|
|
|
|
thereby alleviating this problem and making the "run" scripts much more
|
|
|
|
readable as a bonus.
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
So far we have solutions for shell scripts.
|
|
|
|
What about Python tasks?
|
|
|
|
The same approach applies even though these tasks are more difficult.
|
|
|
|
The process needs to figure out what variables a Python function accesses
|
|
|
|
and what functions it calls.
|
|
|
|
Again, the incremental build solution contains code that first figures out
|
|
|
|
the variable and function dependencies, and then creates a checksum for the data
|
|
|
|
used as the input to the task.
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
Like the working directory case, situations exist where dependencies
|
|
|
|
should be ignored.
|
|
|
|
For these cases, you can instruct the build process to ignore a dependency
|
|
|
|
by using a line like the following:
|
|
|
|
<literallayout class='monospaced'>
|
|
|
|
PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
|
|
|
|
</literallayout>
|
|
|
|
This example ensures that the <filename>PACKAGE_ARCHS</filename> variable does not
|
|
|
|
depend on the value of <filename>MACHINE</filename>, even if it does reference it.
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
|
|
|
|
2014-02-18 20:46:13 +00:00
|
|
|
<para>
|
|
|
|
Equally, there are cases where we need to add dependencies BitBake
|
|
|
|
is not able to find.
|
|
|
|
You can accomplish this by using a line like the following:
|
|
|
|
<literallayout class='monospaced'>
|
|
|
|
PACKAGE_ARCHS[vardeps] = "MACHINE"
|
|
|
|
</literallayout>
|
|
|
|
This example explicitly adds the <filename>MACHINE</filename> variable as a
|
|
|
|
dependency for <filename>PACKAGE_ARCHS</filename>.
|
|
|
|
</para>
|
2014-02-14 19:55:42 +00:00
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
Consider a case with in-line Python, for example, where BitBake is not
|
|
|
|
able to figure out dependencies.
|
|
|
|
When running in debug mode (i.e. using <filename>-DDD</filename>), BitBake
|
|
|
|
produces output when it discovers something for which it cannot figure out
|
|
|
|
dependencies.
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
Thus far, this section has limited discussion to the direct inputs into a task.
|
|
|
|
Information based on direct inputs is referred to as the "basehash" in the
|
|
|
|
code.
|
|
|
|
However, there is still the question of a task's indirect inputs - the
|
|
|
|
things that were already built and present in the build directory.
|
|
|
|
The checksum (or signature) for a particular task needs to add the hashes
|
|
|
|
of all the tasks on which the particular task depends.
|
|
|
|
Choosing which dependencies to add is a policy decision.
|
|
|
|
However, the effect is to generate a master checksum that combines the basehash
|
|
|
|
and the hashes of the task's dependencies.
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
At the code level, there are a variety of ways both the basehash and the
|
|
|
|
dependent task hashes can be influenced.
|
|
|
|
Within the BitBake configuration file, we can give BitBake some extra information
|
|
|
|
to help it construct the basehash.
|
|
|
|
The following statement effectively results in a list of global variable
|
|
|
|
dependency excludes - variables never included in any checksum.
|
|
|
|
This example uses variables from OpenEmbedded to help illustrate
|
|
|
|
the concept:
|
|
|
|
<literallayout class='monospaced'>
|
|
|
|
BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
|
|
|
|
SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
|
|
|
|
USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
|
|
|
|
PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
|
|
|
|
CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
|
|
|
|
</literallayout>
|
|
|
|
The previous example excludes the work directory, which is part of
|
|
|
|
<filename>TMPDIR</filename>.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The rules for deciding which hashes of dependent tasks to include through
|
|
|
|
dependency chains are more complex and are generally accomplished with a
|
|
|
|
Python function.
|
|
|
|
The code in <filename>meta/lib/oe/sstatesig.py</filename> shows two examples
|
|
|
|
of this and also illustrates how you can insert your own policy into the system
|
|
|
|
if so desired.
|
|
|
|
This file defines the two basic signature generators OpenEmbedded Core
|
|
|
|
uses: "OEBasic" and "OEBasicHash".
|
|
|
|
By default, there is a dummy "noop" signature handler enabled in BitBake.
|
|
|
|
This means that behavior is unchanged from previous versions.
|
|
|
|
<filename>OE-Core</filename> uses the "OEBasicHash" signature handler by default
|
|
|
|
through this setting in the <filename>bitbake.conf</filename> file:
|
|
|
|
<literallayout class='monospaced'>
|
|
|
|
BB_SIGNATURE_HANDLER ?= "OEBasicHash"
|
|
|
|
</literallayout>
|
|
|
|
The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> is the same as the
|
|
|
|
"OEBasic" version but adds the task hash to the stamp files.
|
|
|
|
This results in any metadata change that changes the task hash, automatically
|
|
|
|
causing the task to be run again.
|
|
|
|
This removes the need to bump
|
|
|
|
<link linkend='var-PR'><filename>PR</filename></link>
|
|
|
|
values, and changes to metadata automatically ripple across the build.
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
It is also worth noting that the end result of these signature generators is to
|
|
|
|
make some dependency and hash information available to the build.
|
|
|
|
This information includes:
|
|
|
|
<itemizedlist>
|
|
|
|
<listitem><para><filename>BB_BASEHASH_task-<taskname></filename>:
|
|
|
|
The base hashes for each task in the recipe.
|
|
|
|
</para></listitem>
|
|
|
|
<listitem><para><filename>BB_BASEHASH_<filename:taskname></filename>:
|
|
|
|
The base hashes for each dependent task.
|
|
|
|
</para></listitem>
|
|
|
|
<listitem><para><filename>BBHASHDEPS_<filename:taskname></filename>:
|
|
|
|
The task dependencies for each task.
|
|
|
|
</para></listitem>
|
|
|
|
<listitem><para><filename>BB_TASKHASH</filename>:
|
|
|
|
The hash of the currently running task.
|
|
|
|
</para></listitem>
|
|
|
|
</itemizedlist>
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2014-02-18 20:46:13 +00:00
|
|
|
You can find more information on checksum metadata in the
|
|
|
|
"<link linkend='task-checksums-and-setscene'>Task Checksums and Setscene</link>"
|
|
|
|
section.
|
2014-02-14 19:55:42 +00:00
|
|
|
</para>
|
2014-02-11 19:45:54 +00:00
|
|
|
</section>
|
|
|
|
</chapter>
|