sysmo-bts
/
generic-poky
9
0
Fork 0
Code Pull Requests Releases Activity

documenation/poky-ref-manual/technical-details.xml: Shared State 

I updated the tips and tricks section to include two sub-sections.
The first if the debugging stuff from Richard's email.  The
second section is how to invalidate a section of the sstate
cache on a per-class basis.

Fixes [YOCTO #1500]

(From yocto-docs rev: e2cc31c112fc55c3f793f3c416311a1d317ceb37)

Signed-off-by: Scott Rifenbark <scott.m.rifenbark@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is contained in:
Scott Rifenbark 2011-12-14 10:26:17 -08:00 committed by Richard Purdie
parent 8add7fccde
commit 43fb63af31
1 changed files with 73 additions and 98 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

171
documentation/poky-ref-manual/technical-details.xml
View File

@ -493,111 +493,86 @@
<para>
The code in the Yocto Project that supports incremental builds is not
simple code.
Consequently, when things go wrong, debugging needs to be straightforward.
Because of this, the Yocto Project team included strong debugging
tools.
simple code.
This section presents some tips and tricks that help you work around
issues related to shared state code.
</para>
<para>
First, whenever a shared state package is written out, so is a
corresponding <filename>.siginfo</filename> file.
This practice results in a pickled python database of all
the metadata that went into creating the hash for a given shared state
package.
</para>
<section id='debugging'>
<title>Debugging</title>
<para>
Second, if BitBake is run with the <filename>--dump-signatures</filename>
(or <filename>-S</filename>) option, BitBake dumps out
<filename>.siginfo</filename> files in
the stamp directory for every task it would have executed instead of
building the target package specified.
</para>
<para>
When things go wrong, debugging needs to be straightforward.
Because of this, the Yocto Project team included strong debugging
tools:
<itemizedlist>
<listitem><para>Whenever a shared state package is written out, so is a
corresponding <filename>.siginfo</filename> file.
This practice results in a pickled python database of all
the metadata that went into creating the hash for a given shared state
package.</para></listitem>
<listitem><para>If BitBake is run with the <filename>--dump-signatures</filename>
(or <filename>-S</filename>) option, BitBake dumps out
<filename>.siginfo</filename> files in
the stamp directory for every task it would have executed instead of
building the target package specified.</para></listitem>
<listitem><para>There is a <filename>bitbake-diffsigs</filename> command that
can process these <filename>.siginfo</filename> files.
If one file is specified, it will dump out the dependency
information in the file.
If two files are specified, it will compare the
two files and dump out the differences between the two.
This allows the question of "What changed between X and Y?" to be
answered easily.</para></listitem>
</itemizedlist>
</para>
</section>
<para>
Finally, there is a <filename>bitbake-diffsigs</filename> command that
can process these <filename>.siginfo</filename> files.
If one file is specified, it will dump out the dependency
information in the file.
If two files are specified, it will compare the
two files and dump out the differences between the two.
This allows the question of "What changed between X and Y?" to be
answered easily.
</para>
<section id='invalidating-shared-state'>
<title>Invalidating Shared State</title>
<para>
The Yocto Project shared state code uses checksums and shared state memory
cache to avoide building tasks that don't need built.
As with all schemes, this one has some drawbacks.
It is possible that you could make implicit changes that are not factored into the checksum
calculation, but do affect a task's output.
A good example is perhaps when a tool changes its output.
Let's say that the output of <filename>rpmdeps</filename> needed to change.
The result of the change should be that all the "package", "package_write_rpm",
and "package_deploy-rpm" shared state cache items would become invalid.
But, because this is a change that is external to the code and therefore implicit,
the associated shared state cache items do not become invalidated.
In this case, the build process would use the cache items rather than running the
task again.
Obviously, these types of implicit changes can cause problems.
</para>
<para>
To avoid these problems during the build, you need to understand the effects of any
change you make.
Note that any changes you make directly to a function automatically are factored into
the checksum calculation and thus, will invalidate the associated area of sstate cache.
You need to be aware of any implicit changes that are not obvious changes to the
code and could affect the output of a given task.
Once you are aware of such a change, you can take steps to invalidate the cache
and force the task to run.
The step to take is as simple as changing a function's comments in the source code.
For example, to invalidate package shared state files, change the comment statments
of <filename>do_package</filename> or the comments of one of the functions it calls.
The change is purely cosmetic, but it causes the checksum to be recalculated and
forces the task to be run again.
</para>
<note>
For an example of a commit that makes a cosmetic change to invalidate an sstate,
see this
<ulink url='http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
</note>
</section>
</section>
</section>
<!--
<para>
The Yocto Project build process uses a shared state caching scheme to avoid having to
rebuild software when it is not necessary.
Because the build time for a Yocto image can be significant, it is helpful to try and
determine what really needs built and what can be skipped given a particular project's
build process.
</para>
<para>
The scheme that the Yocto Project uses involves checksum generation and comparison for
a task's inputs.
The scheme also employs an area of memory called the shared state cache that is
pointed to by the <filename>SSTATE_DIR</filename> variable.
This area contains task output generated from a previous build.
If a given task's checksum matches the checksum of a previous build for the same
task, the build process uses the state of the cache rather than rerunning that
task.
</para>
<para>
The previous paragraph is a simplistic explanation of how the build process
uses checksums and shared state memory cache to avoide building tasks that
don't need built.
If you want a bit more explanation on the topic,
see "<ulink url='https://lists.yoctoproject.org/pipermail/yocto/2011-March/003366.html'>Shared
State - What does it mean and why should I care?</ulink>" from the Yocto
Project discussion archives.
</para>
<para>
As with all schemes, this one has some drawbacks.
It is possible that you could make implicit changes that are not factored into the checksum
calculation, but do affect a task's output.
A good example is perhaps when a tool changes its output.
Let's say that the output of <filename>rpmdeps</filename> needed to change.
The result of the change should be that all the "package", "package_write_rpm",
and "package_deploy-rpm" sstate-cache items would become invalid.
But, because this is a change that is external to the code and therefore implicit,
the associated sstate-cache items do not become invalidated.
In this case, the build process would use the cache items rather than running the
task again.
Obviously, these types of implicit changes can cause problems.
</para>
<para>
To avoid these problems during the build, you need to understand the effects of any
change you make.
Note that any changes you make directly to a function automatically are factored into
the checksum calculation and thus, will invalidate the associated area of sstate cache.
You need to be aware of any implicit changes that are not obvious changes to the
code and could affect the output of a given task.
Once you are aware of such a change, you can take steps to invalidate the cache
and force the task to run.
The step to take is as simple as changing a function's comments in the source code.
For example, to invalidate package sstate files, change the comment statments
of <filename>do_package</filename> or one of the functions it calls.
The change is purely cosmetic, but it causes the checksum to be recalculated and
forces the task to be run again.
</para>
<note>
For an example of a commit that makes a cosmetic change to invalidate an sstate,
see this
<ulink url='http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
</note>
</section>
-->
</chapter>
<!--
vim: expandtab tw=80 ts=4