documentation/poky-ref-manual/extendpoky.xml: intro changed and order changed

Beefed up the introductory paragraph and I re-ordered the "Making and
Maintaining Changes" section towards the end of the chapter.

(From yocto-docs rev: a58c0e73d720ffb7a4931fbc196ea3831992b514)

Signed-off-by: Scott Rifenbark <scott.m.rifenbark@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is contained in:
Scott Rifenbark 2011-12-08 13:22:23 -08:00 committed by Richard Purdie
parent caab52f6cc
commit d2f1ca8cba
1 changed files with 325 additions and 323 deletions

View File

@ -3,13 +3,14 @@
<chapter id='extendpoky'>
<title>Extending the Yocto Project</title>
<title>Common Tasks</title>
<para>
This chapter provides information about how to extend the functionality
already present in the Yocto Project.
The chapter also documents standard tasks such as adding new
This chapter describes standard tasks such as adding new
software packages, extending or customizing images or porting the Yocto Project to
new hardware (adding a new machine).
The chapter also describes ways to modify package source code, combine multiple
versions of library files into a single image, track license changes, and handle
a package name alias.
Finally, the chapter contains advice about how to make changes to the
Yocto Project to achieve the best results.
</para>
@ -658,324 +659,6 @@
</section>
</section>
<section id="usingpoky-changes">
<title>Making and Maintaining Changes</title>
<para>
Because the Yocto Project is extremely configurable and flexible,
we recognize that developers will want
to extend, configure or optimize it for their specific uses.
To best keep pace with future Yocto Project changes,
we recommend you make controlled changes to the Yocto Project.
</para>
<para>
The Yocto Project supports a <link linkend='usingpoky-changes-layers'>"layers"</link> concept.
If you use layers properly, you can ease future upgrades and allow segregation
between the Yocto Project core and a given developer's changes.
The following section provides more advice on managing changes to the Yocto Project.
</para>
<section id="usingpoky-changes-layers">
<title>BitBake Layers</title>
<para>
Often, developers want to extend the Yocto Project either by adding packages
or by overriding files contained within the Yocto Project to add their own
functionality.
BitBake has a powerful mechanism called
"layers", which provides a way to handle this extension in a fully
supported and non-invasive fashion.
</para>
<para>
The Yocto Project files include several additional layers such as
<filename>meta-rt</filename> and <filename>meta-yocto</filename>
that demonstrate this functionality.
The <filename>meta-rt</filename> layer is not enabled by default.
However, the <filename>meta-yocto</filename> layer is.
</para>
<para>
To enable a layer, you simply add the layer's path to the
<filename><link linkend='var-BBLAYERS'>BBLAYERS</link></filename> variable in your
<filename>bblayers.conf</filename> file, which is found in the Yocto Project file's
build directory.
The following example shows how to enable the <filename>meta-rt</filename>:
<literallayout class='monospaced'>
LCONF_VERSION = "1"
BBFILES ?= ""
BBLAYERS = " \
/path/to/poky/meta \
/path/to/poky/meta-yocto \
/path/to/poky/meta-rt \
"
</literallayout>
</para>
<para>
BitBake parses each <filename>conf/layer.conf</filename> file for each layer in
<filename>BBLAYERS</filename>
and adds the recipes, classes and configurations contained within the layer to
the Yocto Project.
To create your own layer, independent of the Yocto Project files,
simply create a directory with a <filename>conf/layer.conf</filename> file and
add the directory to your <filename>bblayers.conf</filename> file.
</para>
<para>
The <filename>meta-yocto/conf/layer.conf</filename> file demonstrates the
required syntax:
<literallayout class='monospaced'>
# We have a conf and classes directory, add to BBPATH
BBPATH := "${BBPATH}:${LAYERDIR}"
# We have a packages directory, add to BBFILES
BBFILES := "${BBFILES} ${LAYERDIR}/recipes-*/*/*.bb \
${LAYERDIR}/recipes-*/*/*.bbappend"
BBFILE_COLLECTIONS += "yocto"
BBFILE_PATTERN_yocto := "^${LAYERDIR}/"
BBFILE_PRIORITY_yocto = "5"
</literallayout>
</para>
<para>
In the previous example, the recipes for the layers are added to
<filename><link linkend='var-BBFILES'>BBFILES</link></filename>.
The <filename><link linkend='var-BBFILE_COLLECTIONS'>BBFILE_COLLECTIONS</link></filename>
variable is then appended with the layer name.
The <filename><link linkend='var-BBFILE_PATTERN'>BBFILE_PATTERN</link></filename> variable
immediately expands with a regular expression used to match files from
<filename>BBFILES</filename> into
a particular layer, in this case by using the base pathname.
The <filename><link linkend='var-BBFILE_PRIORITY'>BBFILE_PRIORITY</link></filename> variable
then assigns different priorities to the files in different layers.
Applying priorities is useful in situations where the same package might appear in multiple
layers and allows you to choose what layer should take precedence.
</para>
<para>
Note the use of the <filename><link linkend='var-LAYERDIR'>LAYERDIR</link></filename>
variable with the immediate expansion operator.
The <filename>LAYERDIR</filename> variable expands to the directory of the current layer and
requires the immediate expansion operator so that BitBake does not wait to expand the variable
when it's parsing a different directory.
</para>
<para>
BitBake can locate where other <filename>.bbclass</filename> and configuration files
are applied through the <filename>BBPATH</filename> environment variable.
For these cases, BitBake uses the first file with the matching name found in
<filename>BBPATH</filename>.
This is similar to the way the <filename>PATH</filename> variable is used for binaries.
We recommend, therefore, that you use unique <filename>.bbclass</filename>
and configuration file names in your custom layer.
</para>
<para>
We also recommend the following:
<itemizedlist>
<listitem><para>Store custom layers in a Git repository that uses the
<filename>meta-prvt-XXXX</filename> format.</para></listitem>
<listitem><para>Clone the repository alongside other <filename>meta</filename>
directories in the Yocto Project source files area.</para></listitem>
</itemizedlist>
Following these recommendations keeps your Yocto Project files area and
its configuration entirely inside the Yocto Project's core base.
</para>
</section>
<section id="usingpoky-changes-commits">
<title>Committing Changes</title>
<para>
Modifications to the Yocto Project are often managed under some kind of source
revision control system.
Because some simple practices can significantly improve usability, policy for committing changes
is important.
It helps to use a consistent documentation style when committing changes.
The Yocto Project development team has found the following practices work well:
<itemizedlist>
<listitem><para>The first line of the commit summarizes the change and begins with the
name of the affected package or packages.
However, not all changes apply to specific packages.
Consequently, the prefix could also be a machine name or class name.</para></listitem>
<listitem><para>The second part of the commit (if needed) is a longer more detailed
description of the changes.
Placing a blank line between the first and second parts helps with
readability.</para></listitem>
</itemizedlist>
</para>
<para>
Following is an example commit:
<literallayout class='monospaced'>
bitbake/data.py: Add emit_func() and generate_dependencies() functions
These functions allow generation of dependency data between functions and
variables allowing moves to be made towards generating checksums and allowing
use of the dependency information in other parts of BitBake.
Signed-off-by: Richard Purdie richard.purdie@linuxfoundation.org
</literallayout>
</para>
<para>
All commits should be self-contained such that they leave the
metadata in a consistent state that builds both before and after the
commit is made.
Besides being a good practice to follow, it helps ensure autobuilder test results
are valid.
</para>
</section>
<section id="usingpoky-changes-prbump">
<title>Package Revision Incrementing</title>
<para>
If a committed change results in changing the package output,
then the value of the
<filename><link linkend='var-PR'>PR</link></filename> variable needs to be increased
(or "bumped") as part of that commit.
This means that for new recipes you must be sure to add the <filename>PR</filename>
variable and set its initial value equal to "r0".
Failing to define <filename>PR</filename> makes it easy to miss when you bump a package.
Note that you can only use integer values following the "r" in the
<filename>PR</filename> variable.
</para>
<para>
If you are sharing a common <filename>.inc</filename> file with multiple recipes,
you can also use the
<filename><link linkend='var-INC_PR'>INC_PR</link></filename> variable to ensure that
the recipes sharing the <filename>.inc</filename> file are rebuilt when the
<filename>.inc</filename> file itself is changed.
The <filename>.inc</filename> file must set <filename>INC_PR</filename>
(initially to "r0"), and all recipes referring to it should set <filename>PR</filename>
to "$(INC_PR).0" initially, incrementing the last number when the recipe is changed.
If the <filename>.inc</filename> file is changed then its
<filename>INC_PR</filename> should be incremented.
</para>
<para>
When upgrading the version of a package, assuming the
<filename><link linkend='var-PV'>PV</link></filename> changes,
the <filename>PR</filename> variable should be reset to "r0"
(or "$(INC_PR).0" if you are using <filename>INC_PR</filename>).
</para>
<para>
Usually, version increases occur only to packages.
However, if for some reason <filename>PV</filename> changes but does not
increase, you can increase the
<filename><link linkend='var-PE'>PE</link></filename> variable (Package Epoch).
The <filename>PE</filename> variable defaults to "0".
</para>
<para>
Version numbering strives to follow the
<ulink url='http://www.debian.org/doc/debian-policy/ch-controlfields.html'>
Debian Version Field Policy Guidelines</ulink>.
These guidelines define how versions are compared and what "increasing" a version means.
</para>
<para>
There are two reasons for following the previously mentioned guidelines.
First, to ensure that when a developer updates and rebuilds, they get all the changes to
the repository and do not have to remember to rebuild any sections.
Second, to ensure that target users are able to upgrade their
devices using package manager commands such as <filename>opkg upgrade</filename>
(or similar commands for dpkg/apt or rpm-based systems).
</para>
<para>
The goal is to ensure the Yocto Project has packages that can be upgraded in all cases.
</para>
</section>
<section id="usingpoky-changes-collaborate">
<title>Using The Yocto Project in a Team Environment</title>
<para>
It might not be immediately clear how you can use the Yocto Project in a team environment,
or scale it for a large team of developers.
The specifics of any situation determine the best solution.
Granted that the Yocto Project offers immense flexibility regarding this, practices do exist
that experience has shown work well.
</para>
<para>
The core component of any development effort with the Yocto Project is often an
automated build and testing framework along with an image generation process.
You can use these core components to check that the metadata can be built,
highlight when commits break the build, and provide up-to-date images that
allow developers to test the end result and use it as a base platform for further
development.
Experience shows that buildbot is a good fit for this role.
What works well is to configure buildbot to make two types of builds:
incremental and full (from scratch).
See <ulink url='http://www.yoctoproject.org:8010'>the buildbot for the
Yocto Project</ulink> for an example implementation that uses buildbot.
</para>
<para>
You can tie incremental builds to a commit hook that triggers the build
each time a commit is made to the metadata.
This practice results in useful acid tests that determine whether a given commit
breaks the build in some serious way.
Associating a build to a commit can catch a lot of simple errors.
Furthermore, the tests are fast so developers can get quick feedback on changes.
</para>
<para>
Full builds build and test everything from the ground up.
These types of builds usually happen at predetermined times like during the
night when the machine load is low.
</para>
<para>
Most teams have many pieces of software undergoing active development at any given time.
You can derive large benefits by putting these pieces under the control of a source
control system that is compatible with the Yocto Project (i.e. Git or Subversion (SVN).
You can then set the autobuilder to pull the latest revisions of the packages
and test the latest commits by the builds.
This practice quickly highlights issues.
The Yocto Project easily supports testing configurations that use both a
stable known good revision and a floating revision.
The Yocto Project can also take just the changes from specific source control branches.
This capability allows you to track and test specific changes.
</para>
<para>
Perhaps the hardest part of setting this up is defining the software project or
the Yocto Project metadata policies that surround the different source control systems.
Of course circumstances will be different in each case.
However, this situation reveals one of the Yocto Project's advantages -
the system itself does not
force any particular policy on users, unlike a lot of build systems.
The system allows the best policies to be chosen for the given circumstances.
</para>
</section>
<section id="usingpoky-changes-updatingimages">
<title>Updating Existing Images</title>
<para>
Often, rather than re-flashing a new image, you might wish to install updated
packages into an existing running system.
You can do this by first sharing the <filename>tmp/deploy/ipk/</filename> directory
through a web server and then by changing <filename>/etc/opkg/base-feeds.conf</filename>
to point at the shared server.
Following is an example:
<literallayout class='monospaced'>
$ src/gz all http://www.mysite.com/somedir/deploy/ipk/all
$ src/gz armv7a http://www.mysite.com/somedir/deploy/ipk/armv7a
$ src/gz beagleboard http://www.mysite.com/somedir/deploy/ipk/beagleboard
</literallayout>
</para>
</section>
</section>
<section id="usingpoky-modifing-packages">
<title>Modifying Package Source Code</title>
<para>
@ -1300,7 +983,7 @@
important files that contain the license text for the source code.
It is possible to specify a checksum for an entire file, or a specific section of a
file (specified by beginning and ending line numbers with the "beginline" and "endline"
parameters respectively).
parameters, respectively).
The latter is useful for source files with a license notice header,
README documents, and so forth.
If you do not use the "beginline" parameter, then it is assumed that the text begins on the
@ -1381,6 +1064,325 @@
file found in the Yocto Project files area.
</para>
</section>
<section id="usingpoky-changes">
<title>Making and Maintaining Changes</title>
<para>
Because the Yocto Project is extremely configurable and flexible,
we recognize that developers will want
to extend, configure or optimize it for their specific uses.
To best keep pace with future Yocto Project changes,
we recommend you make controlled changes to the Yocto Project.
</para>
<para>
The Yocto Project supports a <link linkend='usingpoky-changes-layers'>"layers"</link> concept.
If you use layers properly, you can ease future upgrades and allow segregation
between the Yocto Project core and a given developer's changes.
The following section provides more advice on managing changes to the Yocto Project.
</para>
<section id="usingpoky-changes-layers">
<title>BitBake Layers</title>
<para>
Often, developers want to extend the Yocto Project either by adding packages
or by overriding files contained within the Yocto Project to add their own
functionality.
BitBake has a powerful mechanism called
"layers", which provides a way to handle this extension in a fully
supported and non-invasive fashion.
</para>
<para>
The Yocto Project files include several additional layers such as
<filename>meta-rt</filename> and <filename>meta-yocto</filename>
that demonstrate this functionality.
The <filename>meta-rt</filename> layer is not enabled by default.
However, the <filename>meta-yocto</filename> layer is.
</para>
<para>
To enable a layer, you simply add the layer's path to the
<filename><link linkend='var-BBLAYERS'>BBLAYERS</link></filename> variable in your
<filename>bblayers.conf</filename> file, which is found in the Yocto Project file's
build directory.
The following example shows how to enable the <filename>meta-rt</filename>:
<literallayout class='monospaced'>
LCONF_VERSION = "1"
BBFILES ?= ""
BBLAYERS = " \
/path/to/poky/meta \
/path/to/poky/meta-yocto \
/path/to/poky/meta-rt \
"
</literallayout>
</para>
<para>
BitBake parses each <filename>conf/layer.conf</filename> file for each layer in
<filename>BBLAYERS</filename>
and adds the recipes, classes and configurations contained within the layer to
the Yocto Project.
To create your own layer, independent of the Yocto Project files,
simply create a directory with a <filename>conf/layer.conf</filename> file and
add the directory to your <filename>bblayers.conf</filename> file.
</para>
<para>
The <filename>meta-yocto/conf/layer.conf</filename> file demonstrates the
required syntax:
<literallayout class='monospaced'>
# We have a conf and classes directory, add to BBPATH
BBPATH := "${BBPATH}:${LAYERDIR}"
# We have a packages directory, add to BBFILES
BBFILES := "${BBFILES} ${LAYERDIR}/recipes-*/*/*.bb \
${LAYERDIR}/recipes-*/*/*.bbappend"
BBFILE_COLLECTIONS += "yocto"
BBFILE_PATTERN_yocto := "^${LAYERDIR}/"
BBFILE_PRIORITY_yocto = "5"
</literallayout>
</para>
<para>
In the previous example, the recipes for the layers are added to
<filename><link linkend='var-BBFILES'>BBFILES</link></filename>.
The <filename><link linkend='var-BBFILE_COLLECTIONS'>BBFILE_COLLECTIONS</link></filename>
variable is then appended with the layer name.
The <filename><link linkend='var-BBFILE_PATTERN'>BBFILE_PATTERN</link></filename> variable
immediately expands with a regular expression used to match files from
<filename>BBFILES</filename> into
a particular layer, in this case by using the base pathname.
The <filename><link linkend='var-BBFILE_PRIORITY'>BBFILE_PRIORITY</link></filename> variable
then assigns different priorities to the files in different layers.
Applying priorities is useful in situations where the same package might appear in multiple
layers and allows you to choose what layer should take precedence.
</para>
<para>
Note the use of the <filename><link linkend='var-LAYERDIR'>LAYERDIR</link></filename>
variable with the immediate expansion operator.
The <filename>LAYERDIR</filename> variable expands to the directory of the current layer and
requires the immediate expansion operator so that BitBake does not wait to expand the variable
when it's parsing a different directory.
</para>
<para>
BitBake can locate where other <filename>.bbclass</filename> and configuration files
are applied through the <filename>BBPATH</filename> environment variable.
For these cases, BitBake uses the first file with the matching name found in
<filename>BBPATH</filename>.
This is similar to the way the <filename>PATH</filename> variable is used for binaries.
We recommend, therefore, that you use unique <filename>.bbclass</filename>
and configuration file names in your custom layer.
</para>
<para>
We also recommend the following:
<itemizedlist>
<listitem><para>Store custom layers in a Git repository that uses the
<filename>meta-prvt-XXXX</filename> format.</para></listitem>
<listitem><para>Clone the repository alongside other <filename>meta</filename>
directories in the Yocto Project source files area.</para></listitem>
</itemizedlist>
Following these recommendations keeps your Yocto Project files area and
its configuration entirely inside the Yocto Project's core base.
</para>
</section>
<section id="usingpoky-changes-commits">
<title>Committing Changes</title>
<para>
Modifications to the Yocto Project are often managed under some kind of source
revision control system.
Because some simple practices can significantly improve usability, policy for committing changes
is important.
It helps to use a consistent documentation style when committing changes.
The Yocto Project development team has found the following practices work well:
<itemizedlist>
<listitem><para>The first line of the commit summarizes the change and begins with the
name of the affected package or packages.
However, not all changes apply to specific packages.
Consequently, the prefix could also be a machine name or class name.</para></listitem>
<listitem><para>The second part of the commit (if needed) is a longer more detailed
description of the changes.
Placing a blank line between the first and second parts helps with
readability.</para></listitem>
</itemizedlist>
</para>
<para>
Following is an example commit:
<literallayout class='monospaced'>
bitbake/data.py: Add emit_func() and generate_dependencies() functions
These functions allow generation of dependency data between functions and
variables allowing moves to be made towards generating checksums and allowing
use of the dependency information in other parts of BitBake.
Signed-off-by: Richard Purdie richard.purdie@linuxfoundation.org
</literallayout>
</para>
<para>
All commits should be self-contained such that they leave the
metadata in a consistent state that builds both before and after the
commit is made.
Besides being a good practice to follow, it helps ensure autobuilder test results
are valid.
</para>
</section>
<section id="usingpoky-changes-prbump">
<title>Package Revision Incrementing</title>
<para>
If a committed change results in changing the package output,
then the value of the
<filename><link linkend='var-PR'>PR</link></filename> variable needs to be increased
(or "bumped") as part of that commit.
This means that for new recipes you must be sure to add the <filename>PR</filename>
variable and set its initial value equal to "r0".
Failing to define <filename>PR</filename> makes it easy to miss when you bump a package.
Note that you can only use integer values following the "r" in the
<filename>PR</filename> variable.
</para>
<para>
If you are sharing a common <filename>.inc</filename> file with multiple recipes,
you can also use the
<filename><link linkend='var-INC_PR'>INC_PR</link></filename> variable to ensure that
the recipes sharing the <filename>.inc</filename> file are rebuilt when the
<filename>.inc</filename> file itself is changed.
The <filename>.inc</filename> file must set <filename>INC_PR</filename>
(initially to "r0"), and all recipes referring to it should set <filename>PR</filename>
to "$(INC_PR).0" initially, incrementing the last number when the recipe is changed.
If the <filename>.inc</filename> file is changed then its
<filename>INC_PR</filename> should be incremented.
</para>
<para>
When upgrading the version of a package, assuming the
<filename><link linkend='var-PV'>PV</link></filename> changes,
the <filename>PR</filename> variable should be reset to "r0"
(or "$(INC_PR).0" if you are using <filename>INC_PR</filename>).
</para>
<para>
Usually, version increases occur only to packages.
However, if for some reason <filename>PV</filename> changes but does not
increase, you can increase the
<filename><link linkend='var-PE'>PE</link></filename> variable (Package Epoch).
The <filename>PE</filename> variable defaults to "0".
</para>
<para>
Version numbering strives to follow the
<ulink url='http://www.debian.org/doc/debian-policy/ch-controlfields.html'>
Debian Version Field Policy Guidelines</ulink>.
These guidelines define how versions are compared and what "increasing" a version means.
</para>
<para>
There are two reasons for following the previously mentioned guidelines.
First, to ensure that when a developer updates and rebuilds, they get all the changes to
the repository and do not have to remember to rebuild any sections.
Second, to ensure that target users are able to upgrade their
devices using package manager commands such as <filename>opkg upgrade</filename>
(or similar commands for dpkg/apt or rpm-based systems).
</para>
<para>
The goal is to ensure the Yocto Project has packages that can be upgraded in all cases.
</para>
</section>
<section id="usingpoky-changes-collaborate">
<title>Using The Yocto Project in a Team Environment</title>
<para>
It might not be immediately clear how you can use the Yocto Project in a team environment,
or scale it for a large team of developers.
The specifics of any situation determine the best solution.
Granted that the Yocto Project offers immense flexibility regarding this, practices do exist
that experience has shown work well.
</para>
<para>
The core component of any development effort with the Yocto Project is often an
automated build and testing framework along with an image generation process.
You can use these core components to check that the metadata can be built,
highlight when commits break the build, and provide up-to-date images that
allow developers to test the end result and use it as a base platform for further
development.
Experience shows that buildbot is a good fit for this role.
What works well is to configure buildbot to make two types of builds:
incremental and full (from scratch).
See <ulink url='http://www.yoctoproject.org:8010'>the buildbot for the
Yocto Project</ulink> for an example implementation that uses buildbot.
</para>
<para>
You can tie incremental builds to a commit hook that triggers the build
each time a commit is made to the metadata.
This practice results in useful acid tests that determine whether a given commit
breaks the build in some serious way.
Associating a build to a commit can catch a lot of simple errors.
Furthermore, the tests are fast so developers can get quick feedback on changes.
</para>
<para>
Full builds build and test everything from the ground up.
These types of builds usually happen at predetermined times like during the
night when the machine load is low.
</para>
<para>
Most teams have many pieces of software undergoing active development at any given time.
You can derive large benefits by putting these pieces under the control of a source
control system that is compatible with the Yocto Project (i.e. Git or Subversion (SVN).
You can then set the autobuilder to pull the latest revisions of the packages
and test the latest commits by the builds.
This practice quickly highlights issues.
The Yocto Project easily supports testing configurations that use both a
stable known good revision and a floating revision.
The Yocto Project can also take just the changes from specific source control branches.
This capability allows you to track and test specific changes.
</para>
<para>
Perhaps the hardest part of setting this up is defining the software project or
the Yocto Project metadata policies that surround the different source control systems.
Of course circumstances will be different in each case.
However, this situation reveals one of the Yocto Project's advantages -
the system itself does not
force any particular policy on users, unlike a lot of build systems.
The system allows the best policies to be chosen for the given circumstances.
</para>
</section>
<section id="usingpoky-changes-updatingimages">
<title>Updating Existing Images</title>
<para>
Often, rather than re-flashing a new image, you might wish to install updated
packages into an existing running system.
You can do this by first sharing the <filename>tmp/deploy/ipk/</filename> directory
through a web server and then by changing <filename>/etc/opkg/base-feeds.conf</filename>
to point at the shared server.
Following is an example:
<literallayout class='monospaced'>
$ src/gz all http://www.mysite.com/somedir/deploy/ipk/all
$ src/gz armv7a http://www.mysite.com/somedir/deploy/ipk/armv7a
$ src/gz beagleboard http://www.mysite.com/somedir/deploy/ipk/beagleboard
</literallayout>
</para>
</section>
</section>
</chapter>
<!--