generic-poky/bitbake/doc/user-manual/user-manual-execution.xml

691 lines
33 KiB
XML
Raw Normal View History

<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
<chapter id="user-manual-execution">
<title>Execution</title>
<para>
Tasks can either be a shell task or a Python task.
For shell tasks, BitBake writes a shell script to
<filename>${WORKDIR}/temp/run.do_taskname.pid</filename>
and then executes the script.
The generated shell script contains all the exported variables,
and the shell functions with all variables expanded.
Output from the shell script goes to the file
<filename>${WORKDIR}/temp/log.do_taskname.pid</filename>.
Looking at the expanded shell functions in the run file and
the output in the log files is a useful debugging technique.
</para>
<para>
For Python tasks, BitBake executes the task internally and logs
information to the controlling terminal.
Future versions of BitBake will write the functions to files
similar to the way shell tasks are handled.
Logging will be handled in a way similar to shell tasks as well.
</para>
<para>
Once all the tasks have been completed BitBake exits.
</para>
<section id='parsing-and-execution'>
<title>Parsing and Execution</title>
<section id='parsing-overview'>
<title>Parsing Overview</title>
<para>
BitBake parses configuration files, classes, recipes, and append
files.
</para>
<para>
The first thing BitBake does is look for the
<filename>bitbake.conf</filename> file.
This file resides in the <filename>conf</filename>
directory, which must be listed in
<link linkend='var-BBPATH'><filename>BBPATH</filename></link>.
</para>
<para>
The <filename>bitbake.conf</filename> file lists other configuration
files to include from the <filename>conf</filename> directory below the
directories listed in <filename>BBPATH</filename>.
In general, the most important of these
configuration files from a user's perspective
is <filename>local.conf</filename>, which contains the user's
customized settings for the build environment.
</para>
<para>
Other notable configuration files are the distribution configuration
file and the machine configuration file.
These configuration files are normally identified by
variables unique to the build systems using BitBake.
For example, the Yocto Project uses the
<filename>DISTRO</filename> and <filename>MACHINE</filename>
variables, respectively.
</para>
<para>
After parsing of the configuration files, some standard classes are
included.
The <filename>base.bbclass</filename> file
is always included.
Other classes that are specified in the configuration using the
<link linkend='var-INHERIT'><filename>INHERIT</filename></link>
variable are also included.
BitBake searches for class files in a "classes" subdirectory under
the paths in <filename>BBPATH</filename> in the same way as
configuration files.
</para>
<para>
After classes are included, the variable
<filename>BBFILES</filename> is set, usually in
<filename>local.conf</filename>, and defines the list of
places to search for recipe and append files.
Adding extra content to <filename>BBFILES</filename> is best
achieved through the use of BitBake layers.
</para>
<para>
BitBake parses each recipe and append file located with
<filename>BBFILES</filename> and stores the values of various
variables into the datastore.
In summary, for each recipe and append file pairing, the configuration
plus the base class of variables are set, followed by the data in the
recipe file itself, followed by any inherit commands
that the recipe file might contain.
</para>
<para>
Because parsing recipe and append files is a time consuming
process, a cache, referred to as the "setscene"
is kept to speed up subsequent parsing.
The setscene is invalid if the timestamps of a recipe changes,
any of the include files change, configuration files change,
or class files on which the recipe file depends change.
</para>
</section>
<section id='parsing-configuration-files'>
<title>Configuration files</title>
<para>
Prior to parsing configuration files, Bitbake looks
at certain variables, including:
<itemizedlist>
<listitem><para><link linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link></para></listitem>
<listitem><para><link linkend='var-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link></para></listitem>
<listitem><para><link linkend='var-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link></para></listitem>
<listitem><para><link linkend='var-BB_ORIGENV'><filename>BB_ORIGENV</filename></link></para></listitem>
<listitem><para><link linkend='var-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></link></para></listitem>
<listitem><para><link linkend='var-PREFERRED_PROVIDERS'><filename>PREFERRED_PROVIDERS</filename></link></para></listitem>
</itemizedlist>
</para>
<para>
The first kind of metadata in BitBake is configuration metadata.
This metadata is global, and therefore affects all packages and
tasks that are executed.
</para>
<para>
BitBake will first search the current working directory for an
optional <filename>conf/bblayers.conf</filename> configuration file.
This file is expected to contain a
<link linkend='var-BBLAYERS'><filename>BBLAYERS</filename></link>
variable that is a space delimited list of 'layer' directories.
For each directory in this list, a <filename>conf/layer.conf</filename>
file will be searched for and parsed with the
<link linkend='var-LAYERDIR'><filename>LAYERDIR</filename></link>
variable being set to the directory where the layer was found.
The idea is these files will setup
<link linkend='var-BBPATH'><filename>BBPATH</filename></link>
and other variables correctly for a given build directory automatically
for the user.
</para>
<para>
BitBake will then expect to find <filename>conf/bitbake.conf</filename>
file somewhere in the user specified <filename>BBPATH</filename>.
That configuration file generally has include directives to pull
in any other metadata (generally files specific to architecture,
machine, local and so on).
</para>
<para>
Only variable definitions and include directives are allowed
in <filename>.conf</filename> files.
The following variables include:
<itemizedlist>
<listitem><para>
<link linkend='var-BITBAKE_UI'><filename>BITBAKE_UI</filename></link>
</para></listitem>
<listitem><para>
<link linkend='var-BBDEBUG'><filename>BBDEBUG</filename></link>
</para></listitem>
<listitem><para>
<link linkend='var-MULTI_PROVIDER_WHITELIST'><filename>MULTI_PROVIDER_WHITELIST</filename></link>
</para></listitem>
<listitem><para>
<link linkend='var-BB_NUMBER_PARSE_THREADS'><filename>BB_NUMBER_PARSE_THREADS</filename></link>
</para></listitem>
<listitem><para>
<filename>BBPKGS</filename>
</para></listitem>
<listitem><para>
<link linkend='var-BB_DEFAULT_TASK'><filename>BB_DEFAULT_TASK</filename></link>
</para></listitem>
<listitem><para>
<link linkend='var-TOPDIR'><filename>TOPDIR</filename></link>
</para></listitem>
<listitem><para>
<link linkend='var-BB_VERBOSE_LOGS'><filename>BB_VERBOSE_LOGS</filename></link>
</para></listitem>
<listitem><para>
<link linkend='var-BB_NICE_LEVEL'><filename>BB_NICE_LEVEL</filename></link>
</para></listitem>
<listitem><para>
<link linkend='var-BBFILE_COLLECTIONS'><filename>BBFILE_COLLECTIONS</filename></link>
</para></listitem>
<listitem><para>
<link linkend='var-ASSUME_PROVIDED'><filename>ASSUME_PROVIDED</filename></link>
</para></listitem>
<listitem><para>
<link linkend='var-BB_DANGLINGAPPENDS_WARNONLY'><filename>BB_DANGLINGAPPENDS_WARNONLY</filename></link>
</para></listitem>
<listitem><para>
<link linkend='var-BBINCLUDED'><filename>BBINCLUDED</filename></link>
</para></listitem>
<listitem><para>
<link linkend='var-BBFILE_PRIORITY'><filename>BBFILE_PRIORITY</filename></link>
</para></listitem>
<listitem><para>
<link linkend='var-BUILDNAME'><filename>BUILDNAME</filename></link>
</para></listitem>
<listitem><para>
<link linkend='var-BBMASK'><filename>BBMASK</filename></link>
</para></listitem>
</itemizedlist>
</para>
<section id='layers'>
<title>Layers</title>
<para>
Layers allow you to isolate different types of
customizations from each other.
While you might find it tempting to keep everything in one layer
when working on a single project, the more modular you organize
your metadata, the easier it is to cope with future changes.
</para>
<para>
To illustrate how you can use layers to keep things modular,
consider machine customizations.
These types of customizations typically reside in a special layer,
rather than a general layer, called a Board Specific Package (BSP) Layer.
Furthermore, the machine customizations should be isolated from
recipes and metadata that support a new GUI environment, for
example.
This situation gives you a couple of layers: one for the machine
configurations and one for the GUI environment.
It is important to understand, however, that the BSP layer can still
make machine-specific additions to recipes within
the GUI environment layer without polluting the GUI layer itself
with those machine-specific changes.
You can accomplish this through a recipe that is a BitBake append
(<filename>.bbappend</filename>) file.
</para>
<para>
There are certain variables specific to layers:
<itemizedlist>
<listitem><para>
<link linkend='var-LAYERDEPENDS'><filename>LAYERDEPENDS</filename></link>
</para></listitem>
<listitem><para>
<link linkend='var-LAYERVERSION'><filename>LAYERVERSION</filename></link>
</para></listitem>
</itemizedlist>
</para>
</section>
<section id='schedulers'>
<title>Schedulers</title>
<para>
Variables specific to scheduling functionality exist:
<itemizedlist>
<listitem><para>
<link linkend='var-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
</para></listitem>
<listitem><para>
<link linkend='var-BB_SCHEDULERS'><filename>BB_SCHEDULERS</filename></link>
</para></listitem>
</itemizedlist>
</para>
</section>
</section>
<section id='metadata-classes'>
<title>Classes</title>
<para>
BitBake's rudimentary inheritance mechanism is accomplished using
classes.
As briefly mentioned in the metadata introduction, BitBake
parses a class when an inherit directive is encountered, and it
is located in the <filename>classes</filename> directory
relative to the directories in
<link linkend='var-BBPATH'><filename>BBPATH</filename></link>.
</para>
</section>
<section id='recipe-bb-files'>
<title>Recipe (<filename>.bb</filename>) Files</title>
<para>
Recipe files, which are files that have the
<filename>.bb</filename> file extension, are logical units of
tasks for execution.
Normally, that logical unit is a package that needs to be
built.
</para>
<para>
BitBake obeys all inter-recipe dependencies.
</para>
<para>
Recipe files must reside in locations found in the
<link linkend='var-BBFILES'><filename>BBFILES</filename></link>
variable.
</para>
</section>
<section id='append-bbappend-files'>
<title>Append (<filename>.bbappend</filename>) Files</title>
<para>
Append files, which are files that have the
<filename>.bbappend</filename> file extension, add or
extend build information to an existing
<link linkend='recipe-bb-files'>recipe file</link>.
</para>
<para>
BitBake expects every append file to have a corresponding recipe file.
Furthermore, the append file and corresponding recipe file
must use the same root filename.
The filenames can differ only in the file type suffix used
(e.g. <filename>formfactor_0.0.bb</filename> and
<filename>formfactor_0.0.bbappend</filename>).
</para>
<para>
Information in append files overrides the information in the
similarly-named recipe file.
</para>
</section>
</section>
<section id='bitbake-dev-environment'>
<title>BitBake</title>
<para>
The OpenEmbedded build system uses BitBake to produce images.
BitBake consists of several functional areas.
This section takes a closer look at each of those areas.
</para>
<section id='source-fetching-dev-environment'>
<title>Source Fetching</title>
<para>
The first stages of building a recipe are to fetch and unpack
the source code:
<imagedata fileref="figures/source-fetching.png" align="center" width="6.5in" depth="5in" />
</para>
figures/
<para>
The <filename>do_fetch</filename> and
<filename>do_unpack</filename> tasks fetch the source files
and unpack them into the work directory.
By default, everything is accomplished in the
build directory,
which has a defined structure.
</para>
<para>
Unpacked source files are pointed to by a variable.
For example, in the Yocto Project and OpenEmbedded build systems,
the <filename>S</filename> variable points to these source files.
Each recipe has an area in the Build Directory where the
unpacked source code resides.
The name of that directory for any given recipe is defined from
several different variables.
You can see the variables that define these directories
by looking at the figure that shows the structure and variables
used in the Yocto Project:
<itemizedlist>
<listitem><para><filename>TMPDIR</filename>
</para></listitem>
<listitem><para><filename>PACKAGE_ARCH</filename>
</para></listitem>
<listitem><para><filename>TARGET_OS</filename>
</para></listitem>
<listitem><para><link linkend='var-PN'><filename>PN</filename></link>
</para></listitem>
<listitem><para><link linkend='var-PV'><filename>PV</filename></link>
</para></listitem>
<listitem><para><link linkend='var-PR'><filename>PR</filename></link>
</para></listitem>
<listitem><para><filename>WORKDIR</filename>
</para></listitem>
<listitem><para><filename>S</filename>
</para></listitem>
</itemizedlist>
</para>
<para>
Briefly, the <filename>S</filename> directory contains the
unpacked source files for a recipe.
The <filename>WORKDIR</filename> directory is where all the
building goes on for a given recipe.
</para>
</section>
<section id='patching-dev-environment'>
<title>Patching</title>
<para>
Once source code is fetched and unpacked, BitBake locates
patch files and applies them to the source files:
<imagedata fileref="figures/patching.png" align="center" width="6in" depth="5in" />
</para>
<para>
The <filename>do_patch</filename> task processes recipes by
using the
<link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
variable to locate applicable patch files, which by default
are <filename>*.patch</filename> or
<filename>*.diff</filename> files, or any file if
"apply=yes" is specified for the file in
<filename>SRC_URI</filename>.
</para>
<para>
BitBake finds and applies multiple patches for a single recipe
in the order in which it finds the patches.
Patches are applied to the recipe's source files located in the
<filename>S</filename> directory.
</para>
<para>
For more information on how the source directories are
created, see the
"<link linkend='source-fetching-dev-environment'>Source Fetching</link>"
section.
</para>
</section>
<section id='configuration-and-compilation-dev-environment'>
<title>Configuration and Compilation</title>
<para>
After source code is patched, BitBake executes tasks that
configure and compile the source code:
<imagedata fileref="figures/configuration-compile-autoreconf.png" align="center" width="7in" depth="5in" />
</para>
<para>
This step in the build process consists of three tasks:
<itemizedlist>
<listitem><para><emphasis><filename>do_configure</filename>:</emphasis>
This task configures the source by enabling and
disabling any build-time and configuration options for
the software being built.
Configurations can come from the recipe itself as well
as from an inherited class.
Additionally, the software itself might configure itself
depending on the target for which it is being built.
</para>
<para>The configurations handled by the
<filename>do_configure</filename> task are specific
to source code configuration for the source code
being built by the recipe.</para>
<para>If you are using the Autotools class
(<filename>autotools.bbclass</filename>),
you can add additional configuration options by using
the <filename>EXTRA_OECONF</filename>
variable.
For information on how this variable works within
that class, see the
<filename>meta/classes/autotools.bbclass</filename> file.
</para></listitem>
<listitem><para><emphasis><filename>do_compile</filename>:</emphasis>
Once a configuration task has been satisfied, BitBake
compiles the source using the
<filename>do_compile</filename> task.
Compilation occurs in the directory pointed to by the
<link linkend='var-B'><filename>B</filename></link>
variable.
Realize that the <filename>B</filename> directory is, by
default, the same as the
<filename>S</filename>
directory.</para></listitem>
<listitem><para><emphasis><filename>do_install</filename>:</emphasis>
Once compilation is done, BitBake executes the
<filename>do_install</filename> task.
This task copies files from the <filename>B</filename>
directory and places them in a holding area pointed to
by the <filename>D</filename> variable.</para></listitem>
</itemizedlist>
</para>
</section>
<section id='package-splitting-dev-environment'>
<title>Package Splitting</title>
<para>
After source code is configured and compiled, the
OpenEmbedded build system analyzes
the results and splits the output into packages:
<imagedata fileref="figures/analysis-for-package-splitting.png" align="center" width="7in" depth="7in" />
</para>
<para>
The <filename>do_package</filename> and
<filename>do_packagedata</filename> tasks combine to analyze
the files found in the <filename>D</filename> directory
and split them into subsets based on available packages and
files.
The analyzing process involves the following as well as other
items: splitting out debugging symbols,
looking at shared library dependencies between packages,
and looking at package relationships.
The <filename>do_packagedata</filename> task creates package
metadata based on the analysis such that the
OpenEmbedded build system can generate the final packages.
Working, staged, and intermediate results of the analysis
and package splitting process use these areas:
<itemizedlist>
<listitem><para><filename>PKGD</filename>
</para></listitem>
<listitem><para><filename>PKGDATA_DIR</filename>
</para></listitem>
<listitem><para><filename>PKGDESTWORK</filename>
</para></listitem>
<listitem><para><filename>PKGDEST</filename>
</para></listitem>
</itemizedlist>
The <filename>FILES</filename>
variable defines the files that go into each package in
<link linkend='var-PACKAGES'><filename>PACKAGES</filename></link>.
If you want details on how this is accomplished in the Yocto Project
for example, you can look at the <filename>package.bbclass</filename>
file in a Yocto tree.
</para>
<para>
Depending on the type of packages being created (RPM, DEB, or
IPK), the <filename>do_package_write_*</filename> task
creates the actual packages and places them in the
Package Feed area, which is
<filename>${TMPDIR}/deploy</filename>.
<note>
Support for creating feeds directly from the
<filename>deploy/*</filename> directories does not exist.
Creating such feeds usually requires some kind of feed
maintenance mechanism that would upload the new packages
into an official package feed (e.g. the
Ångström distribution).
This functionality is highly distribution-specific
and thus is not provided out of the box.
</note>
</para>
</section>
<section id='image-generation-dev-environment'>
<title>Image Generation</title>
<para>
Once packages are split and stored in the Package Feeds area,
the OpenEmbedded build system uses BitBake to generate the
root filesystem image:
<imagedata fileref="figures/image-generation.png" align="center" width="6in" depth="7in" />
</para>
<para>
The image generation process consists of several stages and
depends on many variables.
The <filename>do_rootfs</filename> task uses these key variables
to help create the list of packages to actually install:
<itemizedlist>
<listitem><para><filename>IMAGE_INSTALL</filename>:
Lists out the base set of packages to install from
the Package Feeds area.</para></listitem>
<listitem><para><filename>PACKAGE_EXCLUDE</filename>:
Specifies packages that should not be installed.
</para></listitem>
<listitem><para><filename>IMAGE_FEATURES</filename>:
Specifies features to include in the image.
Most of these features map to additional packages for
installation.</para></listitem>
<listitem><para><filename>PACKAGE_CLASSES</filename>:
Specifies the package backend to use and consequently
helps determine where to locate packages within the
Package Feeds area.</para></listitem>
<listitem><para><filename>IMAGE_LINGUAS</filename>:
Determines the language(s) for which additional
language support packages are installed.
</para></listitem>
</itemizedlist>
</para>
<para>
Package installation is under control of the package manager
(e.g. smart/rpm, opkg, or apt/dpkg) regardless of whether or
not package management is enabled for the target.
At the end of the process, if package management is not
enabled for the target, the package manager's data files
are deleted from the root filesystem.
</para>
<para>
During image generation, the build system attempts to run
all post-installation scripts.
Any that fail to run on the build host are run on the
target when the target system is first booted.
If you are using a
read-only root filesystem,
all the post installation scripts must succeed during the
package installation phase since the root filesystem cannot be
written into.
</para>
<para>
During Optimization, optimizing processes are run across
the image.
These processes include <filename>mklibs</filename> and
<filename>prelink</filename>.
The <filename>mklibs</filename> process optimizes the size
of the libraries.
A <filename>prelink</filename> process optimizes the dynamic
linking of shared libraries to reduce start up time of
executables.
</para>
<para>
Part of the image generation process includes compressing the
root filesystem image.
Compression is accomplished through several optimization
routines designed to reduce the overall size of the image.
</para>
<para>
After the root filesystem has been constructed, the image
generation process turns everything into an image file or
a set of image files.
The formats used for the root filesystem depend on the
<filename>IMAGE_FSTYPES</filename> variable.
</para>
<note>
The entire image generation process is run under Pseudo.
Running under Pseudo ensures that the files in the root
filesystem have correct ownership.
</note>
</section>
<section id='sdk-generation-dev-environment'>
<title>SDK Generation</title>
<para>
The OpenEmbedded build system uses BitBake to generate the
Software Development Kit (SDK) installer script:
<imagedata fileref="figures/sdk-generation.png" align="center" width="6in" depth="7in" />
</para>
<para>
Like image generation, the SDK script process consists of
several stages and depends on many variables.
The <filename>do_populate_sdk</filename> task uses these
key variables to help create the list of packages to actually
install.
</para>
<para>
The <filename>do_populate_sdk</filename> task handles two
parts: a target part and a host part.
The target part is the part built for the target hardware and
includes libraries and headers.
The host part is the part of the SDK that runs on the
<filename>SDKMACHINE</filename>.
</para>
<para>
Once both parts are constructed, the
<filename>do_populate_sdk</filename> task performs some cleanup
on both parts.
After the cleanup, the task creates a cross-development
environment setup script and any configuration files that
might be needed.
</para>
<para>
The final output of the task is the Cross-development
toolchain installation script (<filename>.sh</filename> file),
which includes the environment setup script.
</para>
</section>
</section>
</chapter>