sysmo-bts
/
generic-poky
9
0
Fork 0
Code Pull Requests Releases Activity

documentation/poky-ref-manual/usingpoky.xml: partial for YOCTO #1500 

First draft of a re-write to the "Running a Build" section to try
and satisfy YOCTO #1500.  I segmented the section into three areas
rather than a single area.  This allowed me to create a sub-section
for the sstate stuff where it could be addressed on its own.  I sent
the draft out to Richard and Mark H. and got feedback from RP that
is going to cause further changes.  Thus, I am committing this partial
change.

(From yocto-docs rev: f040ed6979e988968863016103aa3ad4e7365159)

Signed-off-by: Scott Rifenbark <scott.m.rifenbark@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is contained in:
Scott Rifenbark 2011-12-07 13:57:53 -08:00 committed by Richard Purdie
parent 9d87cd9952
commit c2494d3014
1 changed files with 233 additions and 47 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

280
documentation/poky-ref-manual/usingpoky.xml
View File

@ -152,67 +152,253 @@
<title>Running a Build</title>
<para>
You can find information on how to build an image using the Yocto Project in the
You can find general information on how to build an image using the
Yocto Project in the
<ulink url='http://www.yoctoproject.org/docs/1.1/yocto-project-qs/yocto-project-qs.html#building-image'>
Building an Image</ulink> section of the
<ulink url='http://www.yoctoproject.org/docs/1.1/yocto-project-qs/yocto-project-qs.html'>
Yocto Project Quick Start</ulink>.
This section provides a quick overview.
This section provides a summary of the build process and provides information
for less obvious aspects of the build process.
</para>
<para>
The first thing you need to do is set up the Yocto Project build environment by sourcing
the environment setup script as follows:
<literallayout class='monospaced'>
$ source oe-init-build-env [build_dir];
</literallayout>
</para>
<section id='build-overview'>
<title>Build Overview</title>
<para>
The <filename>build_dir</filename> is optional and specifies the directory Yocto Project
uses for the build.
If you do not specify a build directory it defaults to <filename>build</filename>
in the Yocto Project files directory structure.
A common practice is to use a different build directory for different targets.
For example, <filename>~/build/x86</filename> for a <filename>qemux86</filename>
target, and <filename>~/build/arm</filename> for a <filename>qemuarm</filename> target.
See <link linkend="structure-core-script">oe-init-build-env</link>
for more information on this script.
</para>
<para>
The first thing you need to do is set up the Yocto Project build environment by sourcing
the environment setup script as follows:
<literallayout class='monospaced'>
$ source oe-init-build-env [build_dir]
</literallayout>
</para>
<para>
Once the Yocto Project build environment is set up, you can build a target using:
<literallayout class='monospaced'>
<para>
The <filename>build_dir</filename> is optional and specifies the directory Yocto Project
uses for the build.
If you do not specify a build directory it defaults to <filename>build</filename>
in your current working directory.
A common practice is to use a different build directory for different targets.
For example, <filename>~/build/x86</filename> for a <filename>qemux86</filename>
target, and <filename>~/build/arm</filename> for a <filename>qemuarm</filename> target.
See <link linkend="structure-core-script">oe-init-build-env</link>
for more information on this script.
</para>
<para>
Once the Yocto Project build environment is set up, you can build a target using:
<literallayout class='monospaced'>
$ bitbake &lt;target&gt;
</literallayout>
</para>
</literallayout>
</para>
<para>
The <filename>target</filename> is the name of the recipe you want to build.
Common targets are the images in <filename>meta/recipes-core/images</filename>,
<filename>/meta/recipes-sato/images</filename>, etc. all found in the Yocto Project
files.
Or, the target can be the name of a recipe for a specific piece of software such as
<application>busybox</application>.
For more details about the images Yocto Project supports, see the
<link linkend="ref-images">'Reference: Images'</link> appendix.
</para>
<para>
The <filename>target</filename> is the name of the recipe you want to build.
Common targets are the images in <filename>meta/recipes-core/images</filename>,
<filename>/meta/recipes-sato/images</filename>, etc. all found in the Yocto Project
files.
Or, the target can be the name of a recipe for a specific piece of software such as
<application>busybox</application>.
For more details about the images Yocto Project supports, see the
<link linkend="ref-images">'Reference: Images'</link> appendix.
</para>
<note>
Building an image without GNU Public License Version 3 (GPLv3) components is
only supported for minimal and base images.
See <link linkend='ref-images'>'Reference: Images'</link> for more information.
</note>
<note>
Building an image without GNU Public License Version 3 (GPLv3) components is
only supported for minimal and base images.
See <link linkend='ref-images'>'Reference: Images'</link> for more information.
</note>
</section>
<note>
When building an image using GPL components, you need to maintain your original
settings and not switch back and forth applying different versions of the GNU
Public License.
If you rebuild using different versions of GPL, dependency errors might occur
due to some components not being rebuilt.
</note>
<section id='building-an-image-using-gpl-components'>
<title>Building an Image Using GPL Components</title>
<para>
When building an image using GPL components, you need to maintain your original
settings and not switch back and forth applying different versions of the GNU
Public License.
If you rebuild using different versions of GPL, dependency errors might occur
due to some components not being rebuilt.
</para>
</section>
<section id="considering-shared-state-cache">
<title>Considering Shared State Cache</title>
<para>
By design, the Yocto Project builds everything from scratch unless it can determine that
a given task's inputs have not changed.
While building from scratch ensures that everything is current, it does also
mean that a lot of time could be spent rebuiding things that don't necessarily need built.
</para>
<para>
The Yocto Project build process uses a shared state caching scheme to avoid having to
rebuild software when it is not necessary.
Because the build time for a Yocto image can be significant, it is helpful to try and
determine what really needs built and what can be skipped given a particular project's
build process.
</para>
<para>
The scheme that the Yocto Project uses involves checksum generation and comparison for
a task's inputs.
The scheme also employs an area of memory called the shared state cache that is
pointed to by the <filename>SSTATE_DIR</filename> variable.
This area contains task output generated from a previous build.
If a given task's checksum matches the checksum of a previous build for the same
task, the build process uses the state of the cache rather than rerunning that
task.
</para>
<para>
The previous paragraph is a simplistic explanation of how the build process
uses checksums and shared state memory cache to avoide building tasks that
don't need built.
If you want a bit more explanation on the topic,
see "<ulink url='https://lists.yoctoproject.org/pipermail/yocto/2011-March/003366.html'>Shared
State - What does it mean and why should I care?</ulink>" from the Yocto
Project discussion archives.
</para>
<para>
As with all schemes, this one has some drawbacks.
It is possible that you could make implicit changes that are not factored into the checksum
calculation, but do affect a task's output.
A good example is perhaps when a tool changes its output.
Let's say that the output of <filename>rpmdeps</filename> needed to change.
The result of the change should be that all the "package", "package_write_rpm",
and "package_deploy-rpm" sstate-cache items would become invalid.
But, because this is a change that is external to the code and therefore implicit,
the associated sstate-cache items do not become invalidated.
In this case, the build process would use the cache items rather than running the
task again.
Obviously, these types of implicit changes can cause problems.
</para>
<para>
To avoid these problems during the build, you need to understand the effects of any
change you make.
Note that any changes you make directly to a function automatically are factored into
the checksum calculation and thus, will invalidate the associated area of sstate cache.
You need to be aware of any implicit changes that are not obvious changes to the
code and could affect the output of a given task.
Once you are aware of such a change, you can take steps to invalidate the cache
and force the task to run.
The step to take is as simple as changing a function's comments in the source code.
For example, to invalidate package sstate files, change the comment statments
of <filename>do_package</filename> or one of the functions it calls.
The change is purely cosmetic, but it causes the checksum to be recalculated and
forces the task to be run again.
</para>
<note>
For an example of a commit that makes a cosmetic change to invalidate an sstate,
see this
<ulink url='http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
</note>
</section>
<!--
<section id="considering-shared-state-cache">
<title>Considering Shared State Cache</title>
<para>
What is shared state in general.
Benefits?
How we handle things
(reference https://lists.yoctoproject.org/pipermail/yocto/2011-March/001157.htm),
which is RP's dissertation on how YP solved it.
We need to talk a bit about checksum generation for tasks and how the
sstate code uses them to figure out what needs rebuilt and what can be re-loaded
from the sstate cache.
Need to tell about cases where an implicit change can mess things up and under
normal situations the state in the sstate cache would be used but it shouldn't be.
This is the scenario described by bug 1500 - typical case.
Then we talk about how we can invalidate parts of the cache on a per-class basis
if needed.
there is a discussion at
https://lists.yoctoproject.org/pipermail/yocto/2011-March/001157.htm
that talks about sstate and how the YP team attacked and solved the problem.
This is probably a good place to get information from to broach the whole
sstate concept.
YP, by default, builds from scratch.
This is good but it means spending a lot of time rebuilding things that don't
necessarily need rebuilding.
The SSTATE_DIR variable points to the directory for the shared state cache that
is used during a build.
A task's inputs have a checksum or signature associated with them.
If the checksum changes on an input as compared to a prior build, the task must be rerun.
The shared state (sstate) code keeps track of what output is generated by which tasks.
So if a task's inputs have not changed then the output associated with the task can
be yanked from some place and re-used. No re-build required for that particular task.
A "run" shell script is created for each task.
You can create a checksum for the task based on the inputs to the task.
When you have this checksum, the code will look at it and compare it to the previous
checksum to see if the task's inputs have changed.
If so, the task needs to be re-run.
Python tasks have python functions that access variables.
Python functions will call other python functions as well.
The solution was to figure out the variable and function dependencies and create
a checksum value for the data coming into the python task.
Here is a conversation with Mark Hatle regarding bug 1500 (638 is related):
(01:23:34 PM) scottrif: mark - you have a minute?
(01:34:05 PM) Mark Hatle: sure..
(01:34:11 PM) Mark Hatle: might be a bit slow to respond, but I'm here
(01:34:45 PM) scottrif: Hi - I am looking at bug 1500 and trying to get a bit of better understanding. Here is the link to the bug - http://bugzilla.pokylinux.org/show_bug.cgi?id=1500
(01:35:25 PM) scottrif: It seems that the key for the user here is to when to "Know" when to put some comments into a function to invalidate certain areas of sstate.
(01:35:49 PM) Mark Hatle: what the issue is, if you make changes to something that is not normally calculated in the checksums for sstate, then you can get package mismatches..
(01:35:50 PM) scottrif: This trick of "knowing"... does it need to be explained?
(01:36:25 PM) Mark Hatle: The developer will have to know they made such a change.. Then to deal with this, they can use a patch like what is referenced to cause specific steps to be invalidated and various steps re-run..
(01:37:01 PM) scottrif: so my question is will the developer know when they make a change like this?
(01:37:04 PM) Mark Hatle: In this case, we change part of the back-end packaging mechanisms.. which changed internal dependency generation. The sstate code does not checksum the internal dependency generation, it assumes that is code that doesn't change behavior
(01:37:24 PM) Mark Hatle: They should understand the ramifications of their changes — and thus know they need to do this.
(01:37:46 PM) Mark Hatle: Examples of times you need to do this. Back end packaging changes occur — i.e. you change the format of dependency generation..
(01:38:38 PM) scottrif: do you have any other examples?
(01:38:39 PM) Mark Hatle: when you change a recipe itself, source code.. it is -not- necessary to do this
(01:38:49 PM) Mark Hatle: RP might be better at examples of when to do it..
(01:39:11 PM) scottrif: right - If I change a recipe then every thing dependent further down the line gets regenerated right?
(01:39:17 PM) Mark Hatle: This should never be necessary when a recipe changes.. it will only be necessary when some classes or back-end (packaging frameworks) change..
(01:39:21 PM) Mark Hatle: ya
(01:39:33 PM) Mark Hatle: Another way to think of this is implicit dependencies..
(01:40:01 PM) Mark Hatle: I change RPM.. If you build something that produces an RPM package.. the assumption is the RPM package won't change, even if the RPM binary changes..
(01:40:10 PM) Mark Hatle: If the format of the package changes.. you would need to do this
(01:40:53 PM) Mark Hatle: RP can probably give you an idea of the various implicit dependencies, and which ones this type of change is needed for
(01:41:26 PM) scottrif: okay. I am struggling a bit with how to word it. what I will do is write something up and send it out to you and RP for a look
(01:41:47 PM) Mark Hatle: ya, I understand.. it's an odd set of situations that can cause this — but we definitely need to document it
(01:42:01 PM) scottrif: I just want the information to help the user understand the conditions when they will want to invalidate parts of the sstate
(01:42:18 PM) scottrif: I will likely use the RPM example as the case to illustrate it
(01:42:26 PM) scottrif: as it seems pretty straight forward
(01:42:28 PM) Mark Hatle: yup. Key thing is it's only needed on implicit dependencies.. Normal case is back end packaging format changes..
(01:42:31 PM) Mark Hatle: yup
(01:42:47 PM) scottrif: ok - thanks Mark
Here is what RP wants to address 1500:
If its desired to change the checksum of a given subset of tasks, maybe
due to a change which isn't directly visible in the code itself (e.g. a
tool changed its output) its possible to do this by changing a function
comments since the sstate checksums include the body of functions. To
invalidate package sstate files for example, do_package or one of the
functions it calls can be changed, even if its just a cosmetic change to
the commends.
http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54
is an example of a commit which does this.
-->
</section>
<section id='usingpoky-install'>
<title>Installing and Using the Result</title>