Require a sig file which SIGGEN_LOCKEDSIGS_i586 is not null, but
the actual SIGGEN_LOCKEDSIGS_i586 is null.
Invoking 'bitbake -e' and we got:
...
3935 # $SIGGEN_LOCKEDSIGS_i586 [2 operations]
3936 # set /path/to/locked-sigs.inc:8576
3938 # del data_smart.py:406 [finalize]
3939 # ""
...
It was caused by the following commit:
...
(Bitbake rev: 899d45b90061eb3cf3e71029072eee42cd80930c)
Author: Richard Purdie <richard.purdie@linuxfoundation.org>
Date: Tue May 31 23:52:50 2011 +0100
bitbake/data_smart: Change overrides behaviour to remove expanded
variables from the datastore
...
We add prefix 't-' to type to workaround the overrides behaviour.
(From OE-Core rev: f6a39cc957bf85ff43513f0b76afc3b2c9c906b6)
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using underscores in the "types" parts of the variable names can cause
unexpected issues with overrides. For example, if you have both
SSTATE_LOCKEDSIGS_x86_64 and SSTATE_LOCKEDSIGS_x86_64_i586, and i586
is in OVERRIDES, then you lose all of the contents in
SSTATE_LOCKEDSIGS_x86_64 and thus don't get some of the locked sstate.
Using '-'s in the variable names instead, eliminates these issues.
(From OE-Core rev: 6662c412a949a9f6b602c848e6303b19db7e5272)
(From OE-Core rev: 65f558a6f762fb13224091dc22903b58eeb9b392)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add code in the sstate hash validation code to ensure it really did
install these from sstate since if it didn't should to warn/abort
the build. The judgment condition is:
1) If a build is replaced by locked sstate-cache, it will triger a
warn/error;
2) If objects are not used from the locked cache, it will triger a
warn/error;
3) Use SIGGEN_LOCKEDSIGS_CHECK_LEVEL variable controls whether this
is just a warning or a fatal error or nothing to report.
[YOCTO #6639]
(From OE-Core rev: 305912dce61c4fed0cbf631aa98a9e6f29db88e4)
(From OE-Core rev: 1683815695f39d4bad352348913f927ac8a1bcf5)
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of a single monolithic SIGGEN_LOCKEDSIGS, split this into
separate variables, one per sstate package architecture. Add in
a new SIGGEN_LOCKEDSIGS_TYPES variable which lists the package
architectures to load in.
SIGGEN_LOCKEDSIGS_TYPES is made machine specific using overrides.
Also sort the hashes in the lists by PN to make diffing them easier.
(From OE-Core rev: d8b0ce35981931a39e7db9d8e78de6e009b34688)
(From OE-Core rev: b42f305ce38b9e0f1a2b7cb9586bbabcd2d27429)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I've been giving things some thought, specifically why sstate doesn't
get used more and why we have people requesting external toolchains. I'm
guessing the issue is that people don't like how often sstate can change
and the lack of an easy way to lock it down.
Locking it down is actually quite easy so patch implements some basics
of how you can do this (for example to a specific toolchain). With an
addition like this to local.conf (or wherever):
SIGGEN_LOCKEDSIGS = "\
gcc-cross:do_populate_sysroot:a8d91b35b98e1494957a2ddaf4598956 \
eglibc:do_populate_sysroot:13e8c68553dc61f9d67564f13b9b2d67 \
eglibc:do_packagedata:bfca0db1782c719d373f8636282596ee \
gcc-cross:do_packagedata:4b601ff4f67601395ee49c46701122f6 \
"
the code at the end of the email will force the hashes to those values
for the recipes mentioned. The system would then find and use those
specific objects from the sstate cache instead of trying to build
anything.
Obviously this is a little simplistic, you might need to put an override
against this to only apply those revisions for a specific architecture
for example. You'd also probably want to put code in the sstate hash
validation code to ensure it really did install these from sstate since
if it didn't you'd want to abort the build.
This patch also implements support to add to bitbake -S which dumps the
locked sstate checksums for each task into a ready prepared include file
locked-sigs.inc (currently placed into cwd). There is a function,
bb.parse.siggen.dump_lockedsigs() which can be called to trigger the
same functionality from task space.
A warning is added to sstate.bbclass through a call back into the siggen
class to warn if objects are not used from the locked cache. The
SIGGEN_ENFORCE_LOCKEDSIGS variable controls whether this is just a warning
or a fatal error.
A script is provided to generate sstate directory from a locked-sigs file.
(From OE-Core rev: 7e14784f2493a19c6bfe3ec3f05a5cf9797a2f22)
(From OE-Core rev: 884d4fa3e77cf32836f14a113c11489076f4a84d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The idea of squashing packagegroup dependencies was to avoid allarch
packages rebuilding upon tune/arch changes. Now that the allarch class
inclusion is conditional, we can narrow down the packagegroup squashing
to be specifically applied to allarch recipes.
(From OE-Core rev: bd8018792155ae842952432900c8431feeaffe30)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its possible sstate symlinks to other sstate mirrors which then my get
removed/cleaned. If we find invalid symlinks, skip over them rather
than error with a backtrace.
(From OE-Core rev: 5ed9bb42abf93aa084dd23ca68cc996a94a51a10)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We want to use the saferecipedeps handling code to allow gcc-cross-* to
work on multiple different tunes. Its currently in target only code
so it needs to be earlier to allow it to work on native-> target
dependencies.
This change has no effect on existing uses but makes gcc-cross become
shared as desired.
(From OE-Core rev: 9e03db2dfab0b534b86fd48c9190b2d7d0d21238)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the renaming of the cross packages, its no longer possible to use
endswith("-cross") and similar to detect cross packages. Replace these
references with other techniques.
This resolves certain build from sstate failures which were due to the
system believing cross packages were target packages and therefore
dependency handling was altered.
(From OE-Core rev: 91edf4cac223298e50a4b8e59dd19f1b272e3418)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was a nasty sstate hash corruption issue occurring where the
fact the testimage bbclass was inherited meant that the checksum
changed due to testimage.bbclass being confused with image.bbclass.
This patch anchors the bbclass names to avoid this confusion.
(From OE-Core rev: 943a75a4f3b6877e4092dae14b59b7afef8cad3d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* I don't have any real evidence or good statistics for this, but when
comparing signature dumps from my big bitbake world builds I usually
see a lot of rebuilds caused by changes in .bbclasses and only very
rare would be the case where oe-core upgrade brings changes in -native
recipes and no change in .bbclasses used from target recipes
* changing the default to include them shouldn't cause significant
increase in rebuilds and sstate reuse a bit safer
* people working on toolchain (e.g. using gcc from AUTOREV) can easily
extend sstate_rundepfilter to ignore them again (it's easier than
removing existing filter), example how add own signature handler in
your layer is here:
9ac3a7c803
(From OE-Core rev: 336a7897e39b9e42dcfcba9e2520ea96b0c6a8d6)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Like fetch, unpack and patch, populate_lic doesn't vary between different
archs so we should mark it as such. This means better sstate cache reuse
with fewer duplicate files as well as less confusing sstate debugging.
sstatesig also needs to account for the fact BPN is used for sstate files
in these cases.
(From OE-Core rev: 3d59d0bed756f64d0092caa3892239c779c4a341)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By resetting filesdates at this point, we lose matches from stamps which
may not have been in sstatedir. When we don't have hashes specicifed,
its better to return all matches and have the caller decide which are
relavent and which are not since this function has no ability to
decide. There will almost always be one match from stamps we need
to keep and refer to.
(From OE-Core rev: f4c1c9ad2c7e944d4926d0629611da97f9df6a9a)
(From OE-Core rev: 8629844640a5a4c6a9d9375e7b637bcbf3451c4c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've dropped sstate-name so we can remove this code. The fallback was
incorrect since we use taskname without the do_ prefix so this patch
updates to account for that too.
(From OE-Core rev: 72ff58124081333d46d37f31f2d1bf40d715e3bd)
(From OE-Core rev: dbc1426ee75bcf2f6d8b18312522a9b823e23173)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the hashes to find isn't specified we need to return matches from both
the sstate cache and the local stamps directory regardless of how many we've
found so far. If we don't do this, we can miss stamps and the comparison is
less accurate/incorrect.
(From OE-Core rev: 08a074e11e2d517b81ca71fd9bda65297bb015a7)
(From OE-Core rev: 04fc682af7c1657b1c9f37b66a78c4ffc8066e24)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the "ABI safe" recipes, we've been excluding those from signatures. This
is fine in the general case but in the specific case of image recipes it breaks.
A good test case is the interfaces file. Editting this causes init-ifupdown
to rebuild but not an image containing it (e.g. core-image-minimal).
We need to ensure the checksums are added to the image recipes and this change
does that.
(From OE-Core rev: fd085f15e7cd093953f974f69277e130174d551d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
packagegroups are allarch and shouldn't change depending on the target
or machine selected. In general they should have good stable namespaces
for their dependencies. As such we can exclude them from rebuilding when
dependency checksums change.
(From OE-Core rev: 80b065ff46322ec0cad039dfd9eb2d010168dba6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are some left over tab characters in the python functions. This
removes them and resolves python 3 errors.
(From OE-Core rev: fafeb381c48291fa65c634c01c244843c8d7fad3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes "bitbake-diffsigs -t" for changes to the stamp directory layout,
and this time uses the actual value of STAMP to get the location of
sigdata files in the stamp directory rather than trying to do it
manually, which should be a little more robust.
(From OE-Core rev: 82412ebabb0f89c694327ae38f7e864ee8511e7f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a function that can be used from BitBake code which will find
signature data (sigdata/siginfo) files based on specified criteria, and
hook it into BitBake as bb.siggen.find_siginfo.
(From OE-Core rev: 9f0453c29891e32f8038c4bbc22ada28bfbf818a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As discussed on the mailing lists, using a suffix to package names is
hard and has lead to many recipes having to do PKGSUFFIX games. Its
looking extremely hard to scale nativesdk much further without hacking
many recipes.
By comparison, using a prefix like multilib does works much better and
doesn't involve "hacking" as many recipes. This change converts nativesdk
to use a prefix using the existing multilib infrastructure.
(From OE-Core rev: 81813c0e322dc04ce4b069117188d8a54dfddb8c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
gcc exposed this issue where cross gcc recipes were not having
same task checksums as libgcc or gcc-runtime the target recipes
which use same shared workdir and it was triggering the unpack
fetch and patch tasks to reexecute and hence the trouble
Now that we have more than 1 package to consider lets combine
the check
Thanks RP for help and this is on the line of patch
793ce6cd9aa632e0f13789c8293770a86085d28d
(From OE-Core rev: 72cc3df204c4b838cc0358a7ac989adfdd5c1760)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we don't do this, target and cross recipes end up with different sstate
checksums for shared work directory tasks which is bad in the case of gcc.
It leads to multiple fetch/unpack tasks against the shared directory
which ends up with build failures/races.
(From OE-Core rev: 793ce6cd9aa632e0f13789c8293770a86085d28d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
its desireable for other layers to be able to append to the list of packages
with 'safe ABI's which are excluded from the sstate signatures.
I can't emphasise enough how careful you need to be with this list, anything
excluded here needs to be things which don't change interface and are consistent
between different machines.
(From OE-Core rev: 5adef35691a956c3071c0a1ed1caf6b58d1ec5a1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding dependencies on machine specific recipes from generic packages
causes a rebuild of the generic package per machine if using signatures
for the stamp files which is unacceptable.
We need to declare that RRECOMMENDS on kernel-module-* are safe
and that we shouldn't care about these machine specific dependencies
from a stamp perspective. This change adds code which does this.
It depends on a change in bitbake to expose the dataCache object
which can be used to make the calculations we need to allow this to
work correctly.
(From OE-Core rev: 91fc672756d45086cdf4e9c6de8e920dcd8cd14e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Where we have machine specific recipes with well defined behaviour, it makes
no sense to rebuild recipes with these as dependencies whenever the machine
changes. This patch lists those well behaved recipes and excludes them from
the task signatures so we can change MACHINE without invalidating existing
PACKAGE_ARCH binaries.
(From OE-Core rev: 07e34778fc74126af1380bf249fd34a5e3df12c2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds SignatureGenerator classes specific to OE. For now,
these emulate the previous behaviour with the exception that
dependencies on quilt-native are now ignored for checksum purposes.
The intent is to allow easier experimentation and customisation of
this code in future as a result of these changes.
Note that these changes require pending bitbake patches.
(From OE-Core rev: cb73cf4299a192e6065d567fae700987c3f937aa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Hongxu Jia
Randy Witt
Richard Purdie
Martin Jansa
Paul Eggleton
Khem Raj