2011-12-08 18:34:44 +00:00
|
|
|
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
|
|
|
|
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
|
|
|
|
<chapter id='technical-details'>
|
|
|
|
<title>Technical Details</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
This chapter provides technical details for various parts of the Yocto Project.
|
|
|
|
Currently, topics include Yocto Project components and shared state (sstate) cache.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<section id='usingpoky-components'>
|
|
|
|
<title>Yocto Project Components</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The BitBake task executor together with various types of configuration files form the
|
|
|
|
Yocto Project core.
|
|
|
|
This section overviews the BitBake task executor and the
|
|
|
|
configuration files by describing what they are used for and how they interact.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
BitBake handles the parsing and execution of the data files.
|
|
|
|
The data itself is of various types:
|
|
|
|
<itemizedlist>
|
|
|
|
<listitem><para><emphasis>Recipes:</emphasis> Provides details about particular
|
|
|
|
pieces of software</para></listitem>
|
|
|
|
<listitem><para><emphasis>Class Data:</emphasis> An abstraction of common build
|
|
|
|
information (e.g. how to build a Linux kernel).</para></listitem>
|
|
|
|
<listitem><para><emphasis>Configuration Data:</emphasis> Defines machine-specific settings,
|
|
|
|
policy decisions, etc.
|
|
|
|
Configuration data acts as the glue to bind everything together.</para></listitem>
|
|
|
|
</itemizedlist>
|
|
|
|
For more information on data, see the
|
|
|
|
<ulink url='http://www.yoctoproject.org/docs/latest/dev-manual/dev-manual.html#yocto-project-terms'>
|
|
|
|
Yocto Project Terms</ulink> section in
|
|
|
|
<ulink url='http://www.yoctoproject.org/docs/latest/dev-manual/dev-manual.html'>
|
|
|
|
The Yocto Project Development Manual</ulink>.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
BitBake knows how to combine multiple data sources together and refers to each data source
|
2011-12-08 21:34:43 +00:00
|
|
|
as a "<link linkend='usingpoky-changes-layers'>layer</link>".
|
2011-12-08 18:34:44 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Following are some brief details on these core components.
|
|
|
|
For more detailed information on these components see the
|
|
|
|
<link linkend='ref-structure'>'Reference: Directory Structure'</link>
|
|
|
|
appendix.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<section id='usingpoky-components-bitbake'>
|
|
|
|
<title>BitBake</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
BitBake is the tool at the heart of the Yocto Project and is responsible
|
|
|
|
for parsing the metadata, generating a list of tasks from it,
|
|
|
|
and then executing those tasks.
|
|
|
|
To see a list of the options BitBake supports, use the following help command:
|
|
|
|
<literallayout class='monospaced'>
|
|
|
|
$ bitbake --help
|
|
|
|
</literallayout>
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The most common usage for BitBake is <filename>bitbake <packagename></filename>, where
|
|
|
|
<filename>packagename</filename> is the name of the package you want to build
|
|
|
|
(referred to as the "target" in this manual).
|
|
|
|
The target often equates to the first part of a <filename>.bb</filename> filename.
|
|
|
|
So, to run the <filename>matchbox-desktop_1.2.3.bb</filename> file, you
|
|
|
|
might type the following:
|
|
|
|
<literallayout class='monospaced'>
|
|
|
|
$ bitbake matchbox-desktop
|
|
|
|
</literallayout>
|
|
|
|
Several different versions of <filename>matchbox-desktop</filename> might exist.
|
|
|
|
BitBake chooses the one selected by the distribution configuration.
|
|
|
|
You can get more details about how BitBake chooses between different
|
|
|
|
target versions and providers in the
|
|
|
|
<link linkend='ref-bitbake-providers'>Preferences and Providers</link> section.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
BitBake also tries to execute any dependent tasks first.
|
|
|
|
So for example, before building <filename>matchbox-desktop</filename>, BitBake
|
|
|
|
would build a cross compiler and <filename>eglibc</filename> if they had not already
|
|
|
|
been built.
|
|
|
|
<note>This release of the Yocto Project does not support the <filename>glibc</filename>
|
|
|
|
GNU version of the Unix standard C library. By default, the Yocto Project builds with
|
|
|
|
<filename>eglibc</filename>.</note>
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
A useful BitBake option to consider is the <filename>-k</filename> or
|
|
|
|
<filename>--continue</filename> option.
|
|
|
|
This option instructs BitBake to try and continue processing the job as much
|
|
|
|
as possible even after encountering an error.
|
|
|
|
When an error occurs, the target that
|
|
|
|
failed and those that depend on it cannot be remade.
|
|
|
|
However, when you use this option other dependencies can still be processed.
|
|
|
|
</para>
|
|
|
|
</section>
|
|
|
|
|
|
|
|
<section id='usingpoky-components-metadata'>
|
|
|
|
<title>Metadata (Recipes)</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The <filename>.bb</filename> files are usually referred to as "recipes."
|
|
|
|
In general, a recipe contains information about a single piece of software.
|
|
|
|
The information includes the location from which to download the source patches
|
|
|
|
(if any are needed), which special configuration options to apply,
|
|
|
|
how to compile the source files, and how to package the compiled output.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The term "package" can also be used to describe recipes.
|
|
|
|
However, since the same word is used for the packaged output from the Yocto
|
|
|
|
Project (i.e. <filename>.ipk</filename> or <filename>.deb</filename> files),
|
2011-12-08 21:34:43 +00:00
|
|
|
this document avoids using the term "package" when refering to recipes.
|
2011-12-08 18:34:44 +00:00
|
|
|
</para>
|
|
|
|
</section>
|
|
|
|
|
|
|
|
<section id='usingpoky-components-classes'>
|
|
|
|
<title>Classes</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Class files (<filename>.bbclass</filename>) contain information that is useful to share
|
|
|
|
between metadata files.
|
|
|
|
An example is the Autotools class, which contains
|
|
|
|
common settings for any application that Autotools uses.
|
|
|
|
The <link linkend='ref-classes'>Reference: Classes</link> appendix provides details
|
|
|
|
about common classes and how to use them.
|
|
|
|
</para>
|
|
|
|
</section>
|
|
|
|
|
|
|
|
<section id='usingpoky-components-configuration'>
|
|
|
|
<title>Configuration</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The configuration files (<filename>.conf</filename>) define various configuration variables
|
|
|
|
that govern the Yocto Project build process.
|
|
|
|
These files fall into several areas that define machine configuration options,
|
|
|
|
distribution configuration options, compiler tuning options, general common configuration
|
|
|
|
options and user configuration options (<filename>local.conf</filename>, which is found
|
|
|
|
in the Yocto Project files build directory).
|
|
|
|
</para>
|
|
|
|
</section>
|
|
|
|
</section>
|
|
|
|
|
2011-12-08 21:34:43 +00:00
|
|
|
<section id="shared-state-cache">
|
|
|
|
<title>Shared State Cache</title>
|
2011-12-08 18:34:44 +00:00
|
|
|
|
|
|
|
<para>
|
2011-12-15 16:22:30 +00:00
|
|
|
By design, the Yocto Project build system builds everything from scratch unless
|
|
|
|
BitBake can determine that parts don't need to be rebuilt.
|
|
|
|
Fundamentally, building from scratch is attractive as it means all parts are
|
2011-12-13 16:53:45 +00:00
|
|
|
built fresh and there is no possibility of stale data causing problems.
|
|
|
|
When developers hit problems, they typically default back to building from scratch
|
|
|
|
so they know the state of things from the start.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Building an image from scratch is both an advantage and a disadvantage to the process.
|
|
|
|
As mentioned in the previous paragraph, building from scratch ensures that
|
|
|
|
everything is current and starts from a known state.
|
|
|
|
However, building from scratch also takes much longer as it generally means
|
|
|
|
rebuiding things that don't necessarily need rebuilt.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The Yocto Project implements shared state code that supports incremental builds.
|
|
|
|
The implementation of the shared state code answers the following questions that
|
|
|
|
were fundamental roadblocks within the Yocto Project incremental build support system:
|
|
|
|
<itemizedlist>
|
|
|
|
<listitem>What pieces of the system have changed and what pieces have not changed?</listitem>
|
|
|
|
<listitem>How are changed pieces of software removed and replaced?</listitem>
|
|
|
|
<listitem>How are pre-built components that don't need to be rebuilt from scratch
|
|
|
|
used when they are available?</listitem>
|
|
|
|
</itemizedlist>
|
2011-12-08 18:34:44 +00:00
|
|
|
</para>
|
|
|
|
|
2011-12-13 16:53:45 +00:00
|
|
|
<para>
|
|
|
|
For the first question, the build system detects changes in the "inputs" to a given task by
|
|
|
|
creating a checksum (or signature) of the task's inputs.
|
|
|
|
If the checksum changes, the system assumes the inputs have changed and the task needs to be
|
|
|
|
rerun.
|
|
|
|
For the second question, the shared state (sstate) code tracks which tasks add which output
|
|
|
|
to the build process.
|
|
|
|
This means the output from a given task can be removed, upgraded or otherwise manipulated.
|
|
|
|
The third question is partly addressed by the solution for the second question
|
|
|
|
assuming the build system can fetch the sstate objects from remote locations and
|
|
|
|
install them if they are deemed to be valid.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The rest of this section goes into detail about the overall incremental build
|
|
|
|
architecture, the checksums (signatures), shared state, and some tips and tricks.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<section id='overall-architecture'>
|
|
|
|
<title>Overall Architecture</title>
|
|
|
|
|
|
|
|
<para>
|
2011-12-15 16:22:30 +00:00
|
|
|
When determining what parts of the system need to be built, BitBake
|
2011-12-13 16:53:45 +00:00
|
|
|
uses a per-task basis and does not use a per-recipe basis.
|
|
|
|
You might wonder why using a per-task basis is preferred over a per-recipe basis.
|
|
|
|
To help explain, consider having the IPK packaging backend enabled and then switching to DEB.
|
|
|
|
In this case, <filename>do_install</filename> and <filename>do_package</filename>
|
|
|
|
output are still valid.
|
|
|
|
However, with a per-recipe approach, the build would not include the
|
|
|
|
<filename>.deb</filename> files.
|
|
|
|
Consequently, you would have to invalidate the whole build and rerun it.
|
|
|
|
Rerunning everything is not the best situation.
|
|
|
|
Also in this case, the core must be "taught" much about specific tasks.
|
|
|
|
This methodology does not scale well and does not allow users to easily add new tasks
|
|
|
|
in layers or as external recipes without touching the packaged-staging core.
|
|
|
|
</para>
|
|
|
|
</section>
|
|
|
|
|
|
|
|
<section id='checksums'>
|
|
|
|
<title>Checksums (Signatures)</title>
|
|
|
|
|
|
|
|
<para>
|
2011-12-15 16:22:30 +00:00
|
|
|
The shared state code uses a checksum, which is a unique signature of a task's
|
2011-12-13 16:53:45 +00:00
|
|
|
inputs, to determine if a task needs to be run again.
|
2011-12-15 16:22:30 +00:00
|
|
|
Because it is a change in a task's inputs that triggers a rerun, the process
|
2011-12-13 16:53:45 +00:00
|
|
|
needs to detect all the inputs to a given task.
|
|
|
|
For shell tasks, this turns out to be fairly easy because
|
|
|
|
the build process generates a "run" shell script for each task and
|
|
|
|
it is possible to create a checksum that gives you a good idea of when
|
|
|
|
the task's data changes.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
To complicate the problem, there are things that should not be included in
|
|
|
|
the checksum.
|
|
|
|
First, there is the actual specific build path of a given task -
|
|
|
|
the <filename>WORKDIR</filename>.
|
|
|
|
It does not matter if the working directory changes because it should not
|
|
|
|
affect the output for target packages.
|
|
|
|
Also, the build process has the objective of making native/cross packages relocatable.
|
|
|
|
The checksum therefore needs to exclude <filename>WORKDIR</filename>.
|
|
|
|
The simplistic approach for excluding the worknig directory is to set
|
|
|
|
<filename>WORKDIR</filename> to some fixed value and create the checksum
|
|
|
|
for the "run" script.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Another problem results from the "run" scripts containing functions that
|
|
|
|
might or might not get called.
|
2011-12-15 16:22:30 +00:00
|
|
|
The incremental build solution contains code that figures out dependencies
|
|
|
|
between shell functions.
|
2011-12-13 16:53:45 +00:00
|
|
|
This code is used to prune the "run" scripts down to the minimum set,
|
|
|
|
thereby alleviating this problem and making the "run" scripts much more
|
|
|
|
readable as a bonus.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
So far we have solutions for shell scripts.
|
|
|
|
What about python tasks?
|
2011-12-15 16:22:30 +00:00
|
|
|
The same approach applies even though these tasks are more difficult.
|
2011-12-13 16:53:45 +00:00
|
|
|
The process needs to figure out what variables a python function accesses
|
|
|
|
and what functions it calls.
|
2011-12-15 16:22:30 +00:00
|
|
|
Again, the incremental build solution contains code that first figures out
|
|
|
|
the variable and function dependencies, and then creates a checksum for the data
|
|
|
|
used as the input to the task.
|
2011-12-13 16:53:45 +00:00
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Like the <filename>WORKDIR</filename> case, situations exist where dependencies
|
|
|
|
should be ignored.
|
|
|
|
For these cases, you can instruct the build process to ignore a dependency
|
|
|
|
by using a line like the following:
|
|
|
|
<literallayout class='monospaced'>
|
|
|
|
PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
|
|
|
|
</literallayout>
|
|
|
|
This example ensures that the <filename>PACKAGE_ARCHS</filename> variable does not
|
|
|
|
depend on the value of <filename>MACHINE</filename>, even if it does reference it.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2011-12-15 16:22:30 +00:00
|
|
|
Equally, there are cases where we need to add dependencies BitBake is not able to find.
|
2011-12-13 16:53:45 +00:00
|
|
|
You can accomplish this by using a line like the following:
|
|
|
|
<literallayout class='monospaced'>
|
|
|
|
PACKAGE_ARCHS[vardeps] = "MACHINE"
|
|
|
|
</literallayout>
|
|
|
|
This example explicitly adds the <filename>MACHINE</filename> variable as a
|
|
|
|
dependency for <filename>PACKAGE_ARCHS</filename>.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Consider a case with inline python, for example, where BitBake is not
|
|
|
|
able to figure out dependencies.
|
|
|
|
When running in debug mode (i.e. using <filename>-DDD</filename>), BitBake
|
|
|
|
produces output when it discovers something for which it cannot figure out
|
|
|
|
dependencies.
|
|
|
|
The Yocto Project team has currently not managed to cover those dependencies
|
|
|
|
in detail and is aware of the need to fix this situation.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2011-12-15 16:22:30 +00:00
|
|
|
Thus far, this section has limited discussion to the direct inputs into a task.
|
2011-12-13 16:53:45 +00:00
|
|
|
Information based on direct inputs is referred to as the "basehash" in the code.
|
|
|
|
However, there is still the question of a task's indirect inputs, the things that
|
|
|
|
were already built and present in the build directory.
|
|
|
|
The checksum (or signature) for a particular task needs to add the hashes of all the
|
2011-12-15 16:22:30 +00:00
|
|
|
tasks on which the particular task depends.
|
2011-12-13 16:53:45 +00:00
|
|
|
Choosing which dependencies to add is a policy decision.
|
|
|
|
However, the effect is to generate a master checksum that combines the
|
|
|
|
basehash and the hashes of the task's dependencies.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
While figuring out the dependencies and creating these checksums is good,
|
|
|
|
what does the Yocto Project build system do with the checksum information?
|
|
|
|
The build system uses a signature handler that is responsible for
|
|
|
|
processing the checksum information.
|
|
|
|
By default, there is a dummy "noop" signature handler enabled in BitBake.
|
|
|
|
This means that behaviour is unchanged from previous versions.
|
|
|
|
OECore uses the "basic" signature handler through this setting in the
|
|
|
|
<filename>bitbake.conf</filename> file:
|
|
|
|
<literallayout class='monospaced'>
|
|
|
|
BB_SIGNATURE_HANDLER ?= "basic"
|
|
|
|
</literallayout>
|
|
|
|
Also within the BitBake configuration file, we can give BitBake
|
|
|
|
some extra information to help it handle this information.
|
|
|
|
The following statements effectively result in a list of global
|
2011-12-15 16:22:30 +00:00
|
|
|
variable dependency excludes - variables never included in
|
2011-12-13 16:53:45 +00:00
|
|
|
any checksum:
|
|
|
|
<literallayout class='monospaced'>
|
|
|
|
BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH"
|
|
|
|
BB_HASHBASE_WHITELIST += "DL_DIR SSTATE_DIR THISDIR FILESEXTRAPATHS"
|
|
|
|
BB_HASHBASE_WHITELIST += "FILE_DIRNAME HOME LOGNAME SHELL TERM USER"
|
|
|
|
BB_HASHBASE_WHITELIST += "FILESPATH USERNAME STAGING_DIR_HOST STAGING_DIR_TARGET"
|
|
|
|
BB_HASHTASK_WHITELIST += "(.*-cross$|.*-native$|.*-cross-initial$| \
|
|
|
|
.*-cross-intermediate$|^virtual:native:.*|^virtual:nativesdk:.*)"
|
|
|
|
</literallayout>
|
|
|
|
This example is actually where <filename>WORKDIR</filename>
|
|
|
|
is excluded since <filename>WORKDIR</filename> is constructed as a
|
|
|
|
path within <filename>TMPDIR</filename>, which is on the whitelist.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The <filename>BB_HASHTASK_WHITELIST</filename> covers dependent tasks and
|
|
|
|
excludes certain kinds of tasks from the dependency chains.
|
|
|
|
The effect of the previous example is to isolate the native, target,
|
2011-12-15 16:22:30 +00:00
|
|
|
and cross-components.
|
2011-12-13 16:53:45 +00:00
|
|
|
So, for example, toolchain changes do not force a rebuild of the whole system.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The end result of the "basic" handler is to make some dependency and
|
|
|
|
hash information available to the build.
|
|
|
|
This includes:
|
|
|
|
<literallayout class='monospaced'>
|
|
|
|
BB_BASEHASH_task-<taskname> - the base hashes for each task in the recipe
|
|
|
|
BB_BASEHASH_<filename:taskname> - the base hashes for each dependent task
|
|
|
|
BBHASHDEPS_<filename:taskname> - The task dependencies for each task
|
|
|
|
BB_TASKHASH - the hash of the currently running task
|
|
|
|
</literallayout>
|
|
|
|
There is also a "basichash" <filename>BB_SIGNATURE_HANDLER</filename>,
|
|
|
|
which is the same as the basic version but adds the task hash to the stamp files.
|
|
|
|
This results in any metadata change that changes the task hash,
|
|
|
|
automatically causing the task to be run again.
|
|
|
|
This removes the need to bump <filename>PR</filename>
|
|
|
|
values and changes to metadata automatically ripple across the build.
|
|
|
|
Currently, this behavior is not the default behavior.
|
|
|
|
However, it is likely that the Yocto Project team will go forward with this
|
|
|
|
behavior in the future since all the functionality exists.
|
|
|
|
The reason for the delay is the potential impact to the distribution feed
|
|
|
|
creation as they need increasing <filename>PR</filename> fields
|
|
|
|
and the Yocto Project currently lacks a mechanism to automate incrementing
|
|
|
|
this field.
|
|
|
|
</para>
|
|
|
|
</section>
|
|
|
|
|
|
|
|
<section id='shared-state'>
|
|
|
|
<title>Shared State</title>
|
|
|
|
|
|
|
|
<para>
|
2011-12-15 16:22:30 +00:00
|
|
|
Checksums and dependencies, as discussed in the previous section, solve half the
|
2011-12-13 16:53:45 +00:00
|
|
|
problem.
|
|
|
|
The other part of the problem is being able to use checksum information during the build
|
|
|
|
and being able to reuse or rebuild specific components.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The shared state class (<filename>sstate.bbclass</filename>)
|
2011-12-15 16:22:30 +00:00
|
|
|
is a relatively generic implementation of how to "capture" a snapshot of a given task.
|
|
|
|
The idea is that the build process does not care about the source of a task's output.
|
2011-12-13 16:53:45 +00:00
|
|
|
Output could be freshly built or it could be downloaded and unpacked from
|
|
|
|
somewhere - the build process doesn't need to worry about its source.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
There are two types of output, one is just about creating a directory
|
|
|
|
in <filename>WORKDIR</filename>.
|
|
|
|
A good example is the output of either <filename>do_install</filename> or
|
|
|
|
<filename>do_package</filename>.
|
|
|
|
The other type of output occurs when a set of data is merged into a shared directory
|
|
|
|
tree such as the sysroot.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The Yocto Project team has tried to keep the details of the implementation hidden in
|
|
|
|
<filename>sstate.bbclass</filename>.
|
|
|
|
From a user's perspective, adding shared state wrapping to a task
|
|
|
|
is as simple as this <filename>do_deploy</filename> example taken from
|
|
|
|
<filename>do_deploy.bbclass</filename>:
|
|
|
|
<literallayout class='monospaced'>
|
|
|
|
DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
|
|
|
|
SSTATETASKS += "do_deploy"
|
|
|
|
do_deploy[sstate-name] = "deploy"
|
|
|
|
do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"
|
|
|
|
do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
|
|
|
|
|
|
|
|
python do_deploy_setscene () {
|
|
|
|
sstate_setscene(d)
|
|
|
|
}
|
|
|
|
addtask do_deploy_setscene
|
|
|
|
</literallayout>
|
|
|
|
In the example, we add some extra flags to the task, a name field ("deploy"), an
|
|
|
|
input directory where the task sends data, and the output
|
|
|
|
directory where the data from the task should eventually be copied.
|
|
|
|
We also add a <filename>_setscene</filename> variant of the task and add the task
|
|
|
|
name to the <filename>SSTATETASKS</filename> list.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
2011-12-15 16:22:30 +00:00
|
|
|
If you have a directory whose contents you need to preserve, you can do this with
|
|
|
|
a line like the following:
|
2011-12-13 16:53:45 +00:00
|
|
|
<literallayout class='monospaced'>
|
|
|
|
do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
|
|
|
|
</literallayout>
|
|
|
|
This method, as well as the following example, also works for mutliple directories.
|
|
|
|
<literallayout class='monospaced'>
|
|
|
|
do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
|
|
|
|
do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
|
|
|
|
do_package[sstate-lockfile] = "${PACKAGELOCK}"
|
|
|
|
</literallayout>
|
|
|
|
These methods also include the ability to take a lockfile when manipulating
|
|
|
|
shared state directory structures since some cases are sensitive to file
|
|
|
|
additions or removals.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Behind the scenes, the shared state code works by looking in
|
|
|
|
<filename>SSTATE_DIR</filename> and
|
|
|
|
<filename>SSTATE_MIRRORS</filename> for shared state files.
|
|
|
|
Here is an example:
|
|
|
|
<literallayout class='monospaced'>
|
|
|
|
SSTATE_MIRRORS ?= "\
|
|
|
|
file://.* http://someserver.tld/share/sstate/ \n \
|
|
|
|
file://.* file:///some/local/dir/sstate/"
|
|
|
|
</literallayout>
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The shared state package validity can be detected just by looking at the
|
|
|
|
filename since the filename contains the task checksum (or signature) as
|
|
|
|
described earlier in this section.
|
|
|
|
If a valid shared state package is found, the build process downloads it
|
|
|
|
and uses it to accelerate the task.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The build processes uses the <filename>*_setscene</filename> tasks
|
|
|
|
for the task acceleration phase.
|
|
|
|
BitBake goes through this phase before the main execution code and tries
|
|
|
|
to accelerate any tasks for which it can find shared state packages.
|
|
|
|
If a shared state package for a task is available, the shared state
|
|
|
|
package is used.
|
|
|
|
This means the task and any tasks on which it is dependent are not
|
|
|
|
executed.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
As a real world example, the aim is when building an IPK-based image,
|
|
|
|
only the <filename>do_package_write_ipk</filename> tasks would have their
|
|
|
|
shared state packages fetched and extracted.
|
|
|
|
Since the sysroot is not used, it would never get extracted.
|
2011-12-15 16:22:30 +00:00
|
|
|
This is another reason why a task-based approach is preferred over a
|
2011-12-13 16:53:45 +00:00
|
|
|
recipe-based approach, which would have to install the output from every task.
|
|
|
|
</para>
|
|
|
|
</section>
|
|
|
|
|
|
|
|
<section id='tips-and-tricks'>
|
|
|
|
<title>Tips and Tricks</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The code in the Yocto Project that supports incremental builds is not
|
2011-12-14 18:26:17 +00:00
|
|
|
simple code.
|
|
|
|
This section presents some tips and tricks that help you work around
|
|
|
|
issues related to shared state code.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<section id='debugging'>
|
|
|
|
<title>Debugging</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
When things go wrong, debugging needs to be straightforward.
|
|
|
|
Because of this, the Yocto Project team included strong debugging
|
|
|
|
tools:
|
|
|
|
<itemizedlist>
|
|
|
|
<listitem><para>Whenever a shared state package is written out, so is a
|
|
|
|
corresponding <filename>.siginfo</filename> file.
|
|
|
|
This practice results in a pickled python database of all
|
|
|
|
the metadata that went into creating the hash for a given shared state
|
|
|
|
package.</para></listitem>
|
|
|
|
<listitem><para>If BitBake is run with the <filename>--dump-signatures</filename>
|
|
|
|
(or <filename>-S</filename>) option, BitBake dumps out
|
|
|
|
<filename>.siginfo</filename> files in
|
|
|
|
the stamp directory for every task it would have executed instead of
|
2011-12-15 16:22:30 +00:00
|
|
|
building the specified target package.</para></listitem>
|
2011-12-14 18:26:17 +00:00
|
|
|
<listitem><para>There is a <filename>bitbake-diffsigs</filename> command that
|
|
|
|
can process these <filename>.siginfo</filename> files.
|
|
|
|
If one file is specified, it will dump out the dependency
|
|
|
|
information in the file.
|
2011-12-15 16:22:30 +00:00
|
|
|
If two files are specified, it will compare the two files and dump out
|
|
|
|
the differences between the two.
|
2011-12-14 18:26:17 +00:00
|
|
|
This allows the question of "What changed between X and Y?" to be
|
|
|
|
answered easily.</para></listitem>
|
|
|
|
</itemizedlist>
|
|
|
|
</para>
|
|
|
|
</section>
|
|
|
|
|
|
|
|
<section id='invalidating-shared-state'>
|
|
|
|
<title>Invalidating Shared State</title>
|
|
|
|
|
|
|
|
<para>
|
2011-12-15 16:22:30 +00:00
|
|
|
The shared state code uses checksums and shared state memory
|
|
|
|
cache to avoid unnecessarily rebuilding tasks.
|
2011-12-14 18:26:17 +00:00
|
|
|
As with all schemes, this one has some drawbacks.
|
2011-12-15 16:22:30 +00:00
|
|
|
It is possible that you could make implicit changes that are not factored
|
|
|
|
into the checksum calculation, but do affect a task's output.
|
2011-12-14 18:26:17 +00:00
|
|
|
A good example is perhaps when a tool changes its output.
|
|
|
|
Let's say that the output of <filename>rpmdeps</filename> needed to change.
|
|
|
|
The result of the change should be that all the "package", "package_write_rpm",
|
|
|
|
and "package_deploy-rpm" shared state cache items would become invalid.
|
|
|
|
But, because this is a change that is external to the code and therefore implicit,
|
|
|
|
the associated shared state cache items do not become invalidated.
|
2011-12-15 16:22:30 +00:00
|
|
|
In this case, the build process would use the cached items rather than running the
|
2011-12-14 18:26:17 +00:00
|
|
|
task again.
|
|
|
|
Obviously, these types of implicit changes can cause problems.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
To avoid these problems during the build, you need to understand the effects of any
|
|
|
|
change you make.
|
|
|
|
Note that any changes you make directly to a function automatically are factored into
|
|
|
|
the checksum calculation and thus, will invalidate the associated area of sstate cache.
|
|
|
|
You need to be aware of any implicit changes that are not obvious changes to the
|
|
|
|
code and could affect the output of a given task.
|
|
|
|
Once you are aware of such a change, you can take steps to invalidate the cache
|
|
|
|
and force the task to run.
|
|
|
|
The step to take is as simple as changing a function's comments in the source code.
|
|
|
|
For example, to invalidate package shared state files, change the comment statments
|
|
|
|
of <filename>do_package</filename> or the comments of one of the functions it calls.
|
|
|
|
The change is purely cosmetic, but it causes the checksum to be recalculated and
|
|
|
|
forces the task to be run again.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<note>
|
2011-12-15 16:22:30 +00:00
|
|
|
For an example of a commit that makes a cosmetic change to invalidate
|
|
|
|
a shared state, see this
|
2011-12-14 18:26:17 +00:00
|
|
|
<ulink url='http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
|
|
|
|
</note>
|
|
|
|
</section>
|
2011-12-13 16:53:45 +00:00
|
|
|
</section>
|
|
|
|
</section>
|
|
|
|
|
2012-01-04 16:15:21 +00:00
|
|
|
<section id="licenses">
|
|
|
|
<title>Licenses</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
This section describes the mechanism by which the Yocto Project build system
|
|
|
|
tracks changes to licensing text.
|
|
|
|
The section also describes how to enable commercially licensed receipes,
|
|
|
|
which by default are disabled.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<section id="usingpoky-configuring-LIC_FILES_CHKSUM">
|
|
|
|
<title>Tracking License Changes</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The license of an upstream project might change in the future. In order to prevent these changes
|
|
|
|
going unnoticed, the Yocto Project provides a
|
|
|
|
<filename><link linkend='var-LIC_FILES_CHKSUM'>LIC_FILES_CHKSUM</link></filename>
|
|
|
|
variable to track changes to the license text. The checksums are validated at the end of the
|
|
|
|
configure step, and if the checksums do not match, the build will fail.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<section id="usingpoky-specifying-LIC_FILES_CHKSUM">
|
|
|
|
<title>Specifying the <filename>LIC_FILES_CHKSUM</filename> Variable</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The <filename>LIC_FILES_CHKSUM</filename>
|
|
|
|
variable contains checksums of the license text in the source code for the recipe.
|
|
|
|
Following is an example of how to specify <filename>LIC_FILES_CHKSUM</filename>:
|
|
|
|
<literallayout class='monospaced'>
|
|
|
|
LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \
|
|
|
|
file://licfile1.txt;beginline=5;endline=29;md5=yyyy \
|
|
|
|
file://licfile2.txt;endline=50;md5=zzzz \
|
|
|
|
..."
|
|
|
|
</literallayout>
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The Yocto Project uses the
|
|
|
|
<filename><link linkend='var-S'>S</link></filename> variable as the
|
|
|
|
default directory used when searching files listed in
|
|
|
|
<filename>LIC_FILES_CHKSUM</filename>.
|
|
|
|
The previous example employs the default directory.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
You can also use relative paths as shown in the following example:
|
|
|
|
<literallayout class='monospaced'>
|
|
|
|
LIC_FILES_CHKSUM = "file://src/ls.c;startline=5;endline=16;\
|
|
|
|
md5=bb14ed3c4cda583abc85401304b5cd4e"
|
|
|
|
LIC_FILES_CHKSUM = "file://../license.html;md5=5c94767cedb5d6987c902ac850ded2c6"
|
|
|
|
</literallayout>
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
In this example, the first line locates a file in
|
|
|
|
<filename><link linkend='var-S'>S</link>/src/ls.c</filename>.
|
|
|
|
The second line refers to a file in
|
|
|
|
<filename><link linkend='var-WORKDIR'>WORKDIR</link></filename>, which is the parent
|
|
|
|
of <filename>S</filename>.
|
|
|
|
</para>
|
|
|
|
<para>
|
|
|
|
Note that this variable is mandatory for all recipes, unless the
|
|
|
|
<filename>LICENSE</filename> variable is set to "CLOSED".
|
|
|
|
</para>
|
|
|
|
</section>
|
|
|
|
|
|
|
|
<section id="usingpoky-LIC_FILES_CHKSUM-explanation-of-syntax">
|
|
|
|
<title>Explanation of Syntax</title>
|
|
|
|
<para>
|
|
|
|
As mentioned in the previous section, the
|
|
|
|
<filename>LIC_FILES_CHKSUM</filename> variable lists all the
|
|
|
|
important files that contain the license text for the source code.
|
|
|
|
It is possible to specify a checksum for an entire file, or a specific section of a
|
|
|
|
file (specified by beginning and ending line numbers with the "beginline" and "endline"
|
|
|
|
parameters, respectively).
|
|
|
|
The latter is useful for source files with a license notice header,
|
|
|
|
README documents, and so forth.
|
|
|
|
If you do not use the "beginline" parameter, then it is assumed that the text begins on the
|
|
|
|
first line of the file.
|
|
|
|
Similarly, if you do not use the "endline" parameter, it is assumed that the license text
|
|
|
|
ends with the last line of the file.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
The "md5" parameter stores the md5 checksum of the license text.
|
|
|
|
If the license text changes in any way as compared to this parameter
|
|
|
|
then a mismatch occurs.
|
|
|
|
This mismatch triggers a build failure and notifies the developer.
|
|
|
|
Notification allows the developer to review and address the license text changes.
|
|
|
|
Also note that if a mismatch occurs during the build, the correct md5
|
|
|
|
checksum is placed in the build log and can be easily copied to the recipe.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
There is no limit to how many files you can specify using the
|
|
|
|
<filename>LIC_FILES_CHKSUM</filename> variable.
|
|
|
|
Generally, however, every project requires a few specifications for license tracking.
|
|
|
|
Many projects have a "COPYING" file that stores the license information for all the source
|
|
|
|
code files.
|
|
|
|
This practice allows you to just track the "COPYING" file as long as it is kept up to date.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<tip>
|
|
|
|
If you specify an empty or invalid "md5" parameter, BitBake returns an md5 mis-match
|
|
|
|
error and displays the correct "md5" parameter value during the build.
|
|
|
|
The correct parameter is also captured in the build log.
|
|
|
|
</tip>
|
|
|
|
|
|
|
|
<tip>
|
|
|
|
If the whole file contains only license text, you do not need to use the "beginline" and
|
|
|
|
"endline" parameters.
|
|
|
|
</tip>
|
|
|
|
</section>
|
|
|
|
</section>
|
|
|
|
|
|
|
|
<section id="enabling-commercially-licensed-recipes">
|
|
|
|
<title>Enabling Commercially Licensed Recipes</title>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
By default, the Yocto Project build system disables components that
|
|
|
|
have commercial licensing requirements.
|
|
|
|
The following four statements in the
|
|
|
|
<filename>$HOME/poky/meta/conf/distro/poky.conf</filename> file
|
|
|
|
disable components:
|
|
|
|
<literallayout class='monospaced'>
|
|
|
|
COMMERCIAL_LICENSE ?= "lame gst-fluendo-mp3 libmad mpeg2dec ffmpeg qmmp"
|
|
|
|
COMMERCIAL_AUDIO_PLUGINS ?= ""
|
|
|
|
COMMERCIAL_VIDEO_PLUGINS ?= ""
|
|
|
|
COMMERCIAL_QT ?= "qmmp"
|
|
|
|
</literallayout>
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
If you want to enable these components, you can do so by making sure you have
|
|
|
|
the following statements in the configuration file:
|
|
|
|
<literallayout class='monospaced'>
|
|
|
|
COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \
|
|
|
|
gst-plugins-ugly-mpegaudioparse"
|
|
|
|
COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \
|
|
|
|
gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse"
|
|
|
|
COMMERCIAL_LICENSE = ""
|
|
|
|
COMMERCIAL_QT = ""
|
|
|
|
</literallayout>
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Excluding a package name from the
|
|
|
|
<filename>COMMERCIAL_LICENSE</filename> or
|
|
|
|
<filename>COMMERCIAL_QT</filename> statement enables that package.
|
|
|
|
</para>
|
|
|
|
|
|
|
|
<para>
|
|
|
|
Specifying audio and video plug-ins as part of the
|
|
|
|
<filename>COMMERCIAL_AUDIO_PLUGINS</filename> and
|
|
|
|
<filename>COMMERCIAL_VIDEO_PLUGINS</filename> statements includes
|
|
|
|
the plug-ins into built images - thus adding support for media formats.
|
|
|
|
</para>
|
|
|
|
</section>
|
|
|
|
</section>
|
2011-12-08 18:34:44 +00:00
|
|
|
</chapter>
|
|
|
|
<!--
|
|
|
|
vim: expandtab tw=80 ts=4
|
|
|
|
-->
|