documentation/poky-ref-manual/usingpoky.xml: partial for YOCTO #1500
First draft of a re-write to the "Running a Build" section to try and satisfy YOCTO #1500. I segmented the section into three areas rather than a single area. This allowed me to create a sub-section for the sstate stuff where it could be addressed on its own. I sent the draft out to Richard and Mark H. and got feedback from RP that is going to cause further changes. Thus, I am committing this partial change. (From yocto-docs rev: f040ed6979e988968863016103aa3ad4e7365159) Signed-off-by: Scott Rifenbark <scott.m.rifenbark@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is contained in:
parent
9d87cd9952
commit
c2494d3014
|
@ -152,19 +152,24 @@
|
||||||
<title>Running a Build</title>
|
<title>Running a Build</title>
|
||||||
|
|
||||||
<para>
|
<para>
|
||||||
You can find information on how to build an image using the Yocto Project in the
|
You can find general information on how to build an image using the
|
||||||
|
Yocto Project in the
|
||||||
<ulink url='http://www.yoctoproject.org/docs/1.1/yocto-project-qs/yocto-project-qs.html#building-image'>
|
<ulink url='http://www.yoctoproject.org/docs/1.1/yocto-project-qs/yocto-project-qs.html#building-image'>
|
||||||
Building an Image</ulink> section of the
|
Building an Image</ulink> section of the
|
||||||
<ulink url='http://www.yoctoproject.org/docs/1.1/yocto-project-qs/yocto-project-qs.html'>
|
<ulink url='http://www.yoctoproject.org/docs/1.1/yocto-project-qs/yocto-project-qs.html'>
|
||||||
Yocto Project Quick Start</ulink>.
|
Yocto Project Quick Start</ulink>.
|
||||||
This section provides a quick overview.
|
This section provides a summary of the build process and provides information
|
||||||
|
for less obvious aspects of the build process.
|
||||||
</para>
|
</para>
|
||||||
|
|
||||||
|
<section id='build-overview'>
|
||||||
|
<title>Build Overview</title>
|
||||||
|
|
||||||
<para>
|
<para>
|
||||||
The first thing you need to do is set up the Yocto Project build environment by sourcing
|
The first thing you need to do is set up the Yocto Project build environment by sourcing
|
||||||
the environment setup script as follows:
|
the environment setup script as follows:
|
||||||
<literallayout class='monospaced'>
|
<literallayout class='monospaced'>
|
||||||
$ source oe-init-build-env [build_dir];
|
$ source oe-init-build-env [build_dir]
|
||||||
</literallayout>
|
</literallayout>
|
||||||
</para>
|
</para>
|
||||||
|
|
||||||
|
@ -172,7 +177,7 @@
|
||||||
The <filename>build_dir</filename> is optional and specifies the directory Yocto Project
|
The <filename>build_dir</filename> is optional and specifies the directory Yocto Project
|
||||||
uses for the build.
|
uses for the build.
|
||||||
If you do not specify a build directory it defaults to <filename>build</filename>
|
If you do not specify a build directory it defaults to <filename>build</filename>
|
||||||
in the Yocto Project files directory structure.
|
in your current working directory.
|
||||||
A common practice is to use a different build directory for different targets.
|
A common practice is to use a different build directory for different targets.
|
||||||
For example, <filename>~/build/x86</filename> for a <filename>qemux86</filename>
|
For example, <filename>~/build/x86</filename> for a <filename>qemux86</filename>
|
||||||
target, and <filename>~/build/arm</filename> for a <filename>qemuarm</filename> target.
|
target, and <filename>~/build/arm</filename> for a <filename>qemuarm</filename> target.
|
||||||
|
@ -203,16 +208,197 @@
|
||||||
only supported for minimal and base images.
|
only supported for minimal and base images.
|
||||||
See <link linkend='ref-images'>'Reference: Images'</link> for more information.
|
See <link linkend='ref-images'>'Reference: Images'</link> for more information.
|
||||||
</note>
|
</note>
|
||||||
|
</section>
|
||||||
|
|
||||||
<note>
|
<section id='building-an-image-using-gpl-components'>
|
||||||
|
<title>Building an Image Using GPL Components</title>
|
||||||
|
|
||||||
|
<para>
|
||||||
When building an image using GPL components, you need to maintain your original
|
When building an image using GPL components, you need to maintain your original
|
||||||
settings and not switch back and forth applying different versions of the GNU
|
settings and not switch back and forth applying different versions of the GNU
|
||||||
Public License.
|
Public License.
|
||||||
If you rebuild using different versions of GPL, dependency errors might occur
|
If you rebuild using different versions of GPL, dependency errors might occur
|
||||||
due to some components not being rebuilt.
|
due to some components not being rebuilt.
|
||||||
|
</para>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section id="considering-shared-state-cache">
|
||||||
|
<title>Considering Shared State Cache</title>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
By design, the Yocto Project builds everything from scratch unless it can determine that
|
||||||
|
a given task's inputs have not changed.
|
||||||
|
While building from scratch ensures that everything is current, it does also
|
||||||
|
mean that a lot of time could be spent rebuiding things that don't necessarily need built.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
The Yocto Project build process uses a shared state caching scheme to avoid having to
|
||||||
|
rebuild software when it is not necessary.
|
||||||
|
Because the build time for a Yocto image can be significant, it is helpful to try and
|
||||||
|
determine what really needs built and what can be skipped given a particular project's
|
||||||
|
build process.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
The scheme that the Yocto Project uses involves checksum generation and comparison for
|
||||||
|
a task's inputs.
|
||||||
|
The scheme also employs an area of memory called the shared state cache that is
|
||||||
|
pointed to by the <filename>SSTATE_DIR</filename> variable.
|
||||||
|
This area contains task output generated from a previous build.
|
||||||
|
If a given task's checksum matches the checksum of a previous build for the same
|
||||||
|
task, the build process uses the state of the cache rather than rerunning that
|
||||||
|
task.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
The previous paragraph is a simplistic explanation of how the build process
|
||||||
|
uses checksums and shared state memory cache to avoide building tasks that
|
||||||
|
don't need built.
|
||||||
|
If you want a bit more explanation on the topic,
|
||||||
|
see "<ulink url='https://lists.yoctoproject.org/pipermail/yocto/2011-March/003366.html'>Shared
|
||||||
|
State - What does it mean and why should I care?</ulink>" from the Yocto
|
||||||
|
Project discussion archives.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
As with all schemes, this one has some drawbacks.
|
||||||
|
It is possible that you could make implicit changes that are not factored into the checksum
|
||||||
|
calculation, but do affect a task's output.
|
||||||
|
A good example is perhaps when a tool changes its output.
|
||||||
|
Let's say that the output of <filename>rpmdeps</filename> needed to change.
|
||||||
|
The result of the change should be that all the "package", "package_write_rpm",
|
||||||
|
and "package_deploy-rpm" sstate-cache items would become invalid.
|
||||||
|
But, because this is a change that is external to the code and therefore implicit,
|
||||||
|
the associated sstate-cache items do not become invalidated.
|
||||||
|
In this case, the build process would use the cache items rather than running the
|
||||||
|
task again.
|
||||||
|
Obviously, these types of implicit changes can cause problems.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
To avoid these problems during the build, you need to understand the effects of any
|
||||||
|
change you make.
|
||||||
|
Note that any changes you make directly to a function automatically are factored into
|
||||||
|
the checksum calculation and thus, will invalidate the associated area of sstate cache.
|
||||||
|
You need to be aware of any implicit changes that are not obvious changes to the
|
||||||
|
code and could affect the output of a given task.
|
||||||
|
Once you are aware of such a change, you can take steps to invalidate the cache
|
||||||
|
and force the task to run.
|
||||||
|
The step to take is as simple as changing a function's comments in the source code.
|
||||||
|
For example, to invalidate package sstate files, change the comment statments
|
||||||
|
of <filename>do_package</filename> or one of the functions it calls.
|
||||||
|
The change is purely cosmetic, but it causes the checksum to be recalculated and
|
||||||
|
forces the task to be run again.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<note>
|
||||||
|
For an example of a commit that makes a cosmetic change to invalidate an sstate,
|
||||||
|
see this
|
||||||
|
<ulink url='http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
|
||||||
</note>
|
</note>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
|
|
||||||
|
<!--
|
||||||
|
|
||||||
|
<section id="considering-shared-state-cache">
|
||||||
|
<title>Considering Shared State Cache</title>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
What is shared state in general.
|
||||||
|
Benefits?
|
||||||
|
How we handle things
|
||||||
|
(reference https://lists.yoctoproject.org/pipermail/yocto/2011-March/001157.htm),
|
||||||
|
which is RP's dissertation on how YP solved it.
|
||||||
|
We need to talk a bit about checksum generation for tasks and how the
|
||||||
|
sstate code uses them to figure out what needs rebuilt and what can be re-loaded
|
||||||
|
from the sstate cache.
|
||||||
|
Need to tell about cases where an implicit change can mess things up and under
|
||||||
|
normal situations the state in the sstate cache would be used but it shouldn't be.
|
||||||
|
This is the scenario described by bug 1500 - typical case.
|
||||||
|
Then we talk about how we can invalidate parts of the cache on a per-class basis
|
||||||
|
if needed.
|
||||||
|
|
||||||
|
there is a discussion at
|
||||||
|
https://lists.yoctoproject.org/pipermail/yocto/2011-March/001157.htm
|
||||||
|
that talks about sstate and how the YP team attacked and solved the problem.
|
||||||
|
This is probably a good place to get information from to broach the whole
|
||||||
|
sstate concept.
|
||||||
|
|
||||||
|
YP, by default, builds from scratch.
|
||||||
|
This is good but it means spending a lot of time rebuilding things that don't
|
||||||
|
necessarily need rebuilding.
|
||||||
|
|
||||||
|
The SSTATE_DIR variable points to the directory for the shared state cache that
|
||||||
|
is used during a build.
|
||||||
|
|
||||||
|
A task's inputs have a checksum or signature associated with them.
|
||||||
|
If the checksum changes on an input as compared to a prior build, the task must be rerun.
|
||||||
|
The shared state (sstate) code keeps track of what output is generated by which tasks.
|
||||||
|
So if a task's inputs have not changed then the output associated with the task can
|
||||||
|
be yanked from some place and re-used. No re-build required for that particular task.
|
||||||
|
|
||||||
|
A "run" shell script is created for each task.
|
||||||
|
You can create a checksum for the task based on the inputs to the task.
|
||||||
|
When you have this checksum, the code will look at it and compare it to the previous
|
||||||
|
checksum to see if the task's inputs have changed.
|
||||||
|
If so, the task needs to be re-run.
|
||||||
|
|
||||||
|
Python tasks have python functions that access variables.
|
||||||
|
Python functions will call other python functions as well.
|
||||||
|
The solution was to figure out the variable and function dependencies and create
|
||||||
|
a checksum value for the data coming into the python task.
|
||||||
|
|
||||||
|
Here is a conversation with Mark Hatle regarding bug 1500 (638 is related):
|
||||||
|
|
||||||
|
(01:23:34 PM) scottrif: mark - you have a minute?
|
||||||
|
(01:34:05 PM) Mark Hatle: sure..
|
||||||
|
(01:34:11 PM) Mark Hatle: might be a bit slow to respond, but I'm here
|
||||||
|
(01:34:45 PM) scottrif: Hi - I am looking at bug 1500 and trying to get a bit of better understanding. Here is the link to the bug - http://bugzilla.pokylinux.org/show_bug.cgi?id=1500
|
||||||
|
(01:35:25 PM) scottrif: It seems that the key for the user here is to when to "Know" when to put some comments into a function to invalidate certain areas of sstate.
|
||||||
|
(01:35:49 PM) Mark Hatle: what the issue is, if you make changes to something that is not normally calculated in the checksums for sstate, then you can get package mismatches..
|
||||||
|
(01:35:50 PM) scottrif: This trick of "knowing"... does it need to be explained?
|
||||||
|
(01:36:25 PM) Mark Hatle: The developer will have to know they made such a change.. Then to deal with this, they can use a patch like what is referenced to cause specific steps to be invalidated and various steps re-run..
|
||||||
|
(01:37:01 PM) scottrif: so my question is will the developer know when they make a change like this?
|
||||||
|
(01:37:04 PM) Mark Hatle: In this case, we change part of the back-end packaging mechanisms.. which changed internal dependency generation. The sstate code does not checksum the internal dependency generation, it assumes that is code that doesn't change behavior
|
||||||
|
(01:37:24 PM) Mark Hatle: They should understand the ramifications of their changes — and thus know they need to do this.
|
||||||
|
(01:37:46 PM) Mark Hatle: Examples of times you need to do this. Back end packaging changes occur — i.e. you change the format of dependency generation..
|
||||||
|
(01:38:38 PM) scottrif: do you have any other examples?
|
||||||
|
(01:38:39 PM) Mark Hatle: when you change a recipe itself, source code.. it is -not- necessary to do this
|
||||||
|
(01:38:49 PM) Mark Hatle: RP might be better at examples of when to do it..
|
||||||
|
(01:39:11 PM) scottrif: right - If I change a recipe then every thing dependent further down the line gets regenerated right?
|
||||||
|
(01:39:17 PM) Mark Hatle: This should never be necessary when a recipe changes.. it will only be necessary when some classes or back-end (packaging frameworks) change..
|
||||||
|
(01:39:21 PM) Mark Hatle: ya
|
||||||
|
(01:39:33 PM) Mark Hatle: Another way to think of this is implicit dependencies..
|
||||||
|
(01:40:01 PM) Mark Hatle: I change RPM.. If you build something that produces an RPM package.. the assumption is the RPM package won't change, even if the RPM binary changes..
|
||||||
|
(01:40:10 PM) Mark Hatle: If the format of the package changes.. you would need to do this
|
||||||
|
(01:40:53 PM) Mark Hatle: RP can probably give you an idea of the various implicit dependencies, and which ones this type of change is needed for
|
||||||
|
(01:41:26 PM) scottrif: okay. I am struggling a bit with how to word it. what I will do is write something up and send it out to you and RP for a look
|
||||||
|
(01:41:47 PM) Mark Hatle: ya, I understand.. it's an odd set of situations that can cause this — but we definitely need to document it
|
||||||
|
(01:42:01 PM) scottrif: I just want the information to help the user understand the conditions when they will want to invalidate parts of the sstate
|
||||||
|
(01:42:18 PM) scottrif: I will likely use the RPM example as the case to illustrate it
|
||||||
|
(01:42:26 PM) scottrif: as it seems pretty straight forward
|
||||||
|
(01:42:28 PM) Mark Hatle: yup. Key thing is it's only needed on implicit dependencies.. Normal case is back end packaging format changes..
|
||||||
|
(01:42:31 PM) Mark Hatle: yup
|
||||||
|
(01:42:47 PM) scottrif: ok - thanks Mark
|
||||||
|
|
||||||
|
Here is what RP wants to address 1500:
|
||||||
|
|
||||||
|
If its desired to change the checksum of a given subset of tasks, maybe
|
||||||
|
due to a change which isn't directly visible in the code itself (e.g. a
|
||||||
|
tool changed its output) its possible to do this by changing a function
|
||||||
|
comments since the sstate checksums include the body of functions. To
|
||||||
|
invalidate package sstate files for example, do_package or one of the
|
||||||
|
functions it calls can be changed, even if its just a cosmetic change to
|
||||||
|
the commends.
|
||||||
|
http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54
|
||||||
|
is an example of a commit which does this.
|
||||||
|
|
||||||
|
-->
|
||||||
|
</section>
|
||||||
|
|
||||||
|
|
||||||
<section id='usingpoky-install'>
|
<section id='usingpoky-install'>
|
||||||
<title>Installing and Using the Result</title>
|
<title>Installing and Using the Result</title>
|
||||||
|
|
||||||
|
|
Loading…
Reference in New Issue