Commit Graph

106 Commits

Author SHA1 Message Date
Paul Eggleton 5f7bf1f66d bitbake: lib/bb/siggen: show word-diff for single-line values containing spaces
If a variable value has changed and either the new or old value contains
spaces, a word diff should be appropriate and may be a bit more readable.
Import the "simplediff" module and use it to show a word diff (in the
style of GNU wdiff and git diff --word-diff).

Also use a similar style diff to show changes in the runtaskhashes list.
I didn't use an actual word-diff here since it's a little different - we
can be sure that the list is a list and not simply a free-format string.

(Bitbake rev: 20db6b6553c80e18afc4f43dc2495435f7477822)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:32 +01:00
Paul Eggleton 5d8b89fc0b bitbake: lib/bb/siggen: add collapsed mode to compare_sigfiles()
If we just want to drill down to the actual differences then we don't
need to see certain things in the output, e.g. basehash changing or the
signature of dependent tasks. This will be used for comparing signatures
within buildhistory-diff in OE-Core; the default mode as used by
bitbake-diffsigs and bitbake -S printdiff remains unchanged for the
moment.

(Bitbake rev: 6543a59b1ebd3194a7c6421cffc66ebe31a67c62)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:32 +01:00
Paul Eggleton 8c015e45ff bitbake: lib/bb/siggen: don't show unchanged runtaskdeps list
If the runtaskdeps list hasn't actually changed (but the signatures of
some of the tasks did) then it doesn't make sense to print out the old
and new lists as they are both the same and may be very long, e.g. for
do_rootfs in OE.

(Bitbake rev: cb170543605288b3e8badfac3a54c588f4c95413)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:32 +01:00
Paul Eggleton 445e622b23 bitbake: lib/bb/siggen: show a diff when dumping changes to multi-line values
When dumping changes to signatures e.g. output of bitbake -s printdiff,
if for example a function has changed, it's much more readable to see a
unified diff of the changes rather than just printing the old function
followed by the new function, so use difflib to do that.

Note: I elected to keep to one item in the returned list per change,
rather than one line per line of output, so that the caller can still
look at changes individually if needed. Thus I've added some handling to
bitbake-diffsigs to split the change into lines so that each line is
displayed indented.

(Bitbake rev: 4d254ae63a35231c98e3f73f669b040ed1144042)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:32 +01:00
Paul Eggleton f3b010b39a bitbake: lib/bb/siggen: add missing path separator to cleaned paths
Printing "pbzip2pbzip2_1.1.13.bb" is ugly, we need to add a separating
slash so that we get "pbzip2/pbzip2_1.1.13.bb" instead.

(Bitbake rev: 55cd4045a37afc954f4d5091f524756b266064fc)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:32 +01:00
Jianxun Zhang 7eb02e837e bitbake: use multiple processes to dump signatures.
This change significantly shortens the time on reparsing stage
of '-S' option.

Each file is reparsed and then dumped within a dedicated
process. The maximum number of the running processes is not
greater than the value of BB_NUMBER_PARSE_THREADS if it is set.

The dump_sigs() in class SignatureGeneratorBasic is _replaced_
by a new dump_sigfn() interface, so calls from the outside and
subclasses are dispatched to the implementation in the base
class of SignatureGeneratorBasic.

Fixes [YOCTO #10352]

(Bitbake rev: 99d3703edd77a21770b366c6ad65a3c0f5183493)

Signed-off-by: Jianxun Zhang <jianxun.zhang@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-17 11:29:28 +00:00
Richard Purdie d6799e655e bitbake: siggen: Fix clean_basepath to work with multiconfig
Tasknames can now start with "multiconfig:" which broke the virtual: comparison code and
lead to unpredictable checksums with nativesdk recipes. This adds in handling for
the new additional prefix which unbreaks nativesdk builds when using multiconfig.

(Bitbake rev: 4da296c0c93391ec37f8b6e2245071657759771e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-22 12:36:40 +00:00
Joshua Lock 97f83e911e bitbake: bitbake: remove True option to getVar calls (take 2)
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.

Search made with the following regex: getVar ?\(( ?[^,()]*), True\)

(a follow on patch to fix up a few recent introductions)

(Bitbake rev: f9961fd5beb31d5ab9656a5be59f7ab3effef2f0)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:24 +00:00
Paul Eggleton d856bce542 bitbake: siggen: add means of ignoring basehash mismatch
If you run the setVariable command to set variables then you end up
causing the basehash to not match the previously computed values, which
triggers error messages. These mismatches are expected, so add a means
of disabling them.

(Bitbake rev: 5a80c0e210f26526afbe8f266b7b1a9c03334967)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:25:07 +00:00
Joshua Lock 1fce7ecbbb bitbake: bitbake: remove True option to getVar calls
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.

Search made with the following regex: getVar ?\(( ?[^,()]*), True\)

(Bitbake rev: 3b45c479de8640f92dd1d9f147b02e1eecfaadc8)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:09 +00:00
Richard Purdie 589f08aa85 bitbake: siggen: Ensure taskhash mismatches don't override existing data
We recalculate the taskhash to ensure the version we have matches
what we think it should be. When we write out a sigdata file, use
the calculated value so that we don't overwrite any existing file.
This leaves any original taskhash sigdata file intact to allow a
debugging comparison.

(Bitbake rev: 291353b711670ce2da3d45617fc96520bdf09d3f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-04 12:50:55 +00:00
Richard Purdie 0a4a6d6956 bitbake: siggen: Pass basehash to worker processes and sanity check reparsing result
Bitbake can parse metadata in the cooker and in the worker during builds. If
the metadata isn't deterministic, it can change between these two parses and
this confuses things a lot. It turns out to be hard to debug these issues
currently.

This patch ensures the basehashes from the original parsing are passed into
the workers and that these are checked when reparsing for consistency. The user
is shown an error message if inconsistencies are found.

There is debug code in siggen.py (see the "Slow but can be useful for debugging
mismatched basehashes" commented code), we don't enable this by default due to
performance issues. If you run into this message, enable this code and you will
find "sigbasedata" files in tmp/stamps which should correspond to the hashes
shown in this error message. bitbake-diffsigs on the files should show which
variables are changing.

(Bitbake rev: 857829048c14338132784326ba98a71f12192db8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-04 12:50:55 +00:00
Jonathan Liu f078ccf1ac bitbake: siggen: Fix file variable typo in compare_sigfiles
(Bitbake rev: deab9a30987b225922490ca186c5307c15d45b82)

Signed-off-by: Jonathan Liu <net147@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-20 16:11:29 +01:00
Richard Purdie 218b81acb6 bitbake: bitbake: Initial multi-config support
This patch adds the notion of supporting multiple configurations within
a single build. To enable it, set a line in local.conf like:

BBMULTICONFIG = "configA configB configC"

This would tell bitbake that before it parses the base configuration,
it should load conf/configA.conf and so on for each different
configuration. These would contain lines like:

MACHINE = "A"

or other variables which can be set which can be built in the same
build directory (or change TMPDIR not to conflict).

One downside I've already discovered is that if we want to inherit this
file right at the start of parsing, the only place you can put the
configurations is in "cwd", since BBPATH isn't constructed until the
layers are parsed and therefore using it as a preconf file isn't
possible unless its located there.

Execution of these targets takes the form "bitbake
multiconfig:configA:core-image-minimal core-image-sato" so similar to
our virtclass approach for native/nativesdk/multilib using BBCLASSEXTEND.

Implementation wise, the implication is that instead of tasks being
uniquely referenced with "recipename/fn:task" it now needs to be
"configuration:recipename:task".

We already started using "virtual" filenames for recipes when we
implemented BBCLASSEXTEND and this patch adds a new prefix to
these, "multiconfig:<configname>:" and hence avoid changes to a large
part of the codebase thanks to this. databuilder has an internal array
of data stores and uses the right one depending on the supplied virtual
filename.

That trick allows us to use the existing parsing code including the
multithreading mostly unchanged as well as most of the cache code.

For recipecache, we end up with a dict of these accessed by
multiconfig (mc). taskdata and runqueue can only cope with one recipecache
so for taskdata, we pass in each recipecache and have it compute the result
and end up with an array of taskdatas. We can only have one runqueue so there
extensive changes there.

This initial implementation has some drawbacks:

a) There are no inter-multi-configuration dependencies as yet

b) There are no sstate optimisations. This means if the build uses the
same object twice in say two different TMPDIRs, it will either load from
an existing sstate cache at the start or build it twice. We can then in
due course look at ways in which it would only build it once and then
reuse it. This will likely need significant changes to the way sstate
currently works to make that possible.

(Bitbake rev: 5287991691578825c847bac2368e9b51c0ede3f0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18 10:06:27 +01:00
Paul Eggleton fac16ff8f7 bitbake: siggen: properly close files rather than opening them inline
If you don't do this, with Python 3 you get a warning on exit under some
circumstances.

(Bitbake rev: 49502685df3e616023df352823156381b1f79cd3)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18 10:06:26 +01:00
Ulrich Ölmann 5d1d1774ec bitbake: siggen: Fix typo
(Bitbake rev: 40f605199fb3ec2549611508b7576c64d735b2b7)

Signed-off-by: Ulrich Ölmann <u.oelmann@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-11 00:07:09 +01:00
Richard Purdie 0f2c59367a bitbake: bitbake: Convert to python 3
Various misc changes to convert bitbake to python3 which don't warrant
separation into separate commits.

(Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:02 +01:00
Richard Purdie 62384f5467 bitbake: siggen: Fixes to handle sigdata/siginfo files only containing basehash data
The signature data file comparison functions are meant to be able to
handle data files containing just the base hash data. This had regressed
in some places so add fixes to allow these comparisons to be made. The
runtime components in the data files are optional.

(Bitbake rev: 2a6659fd748e255a02c2f9d047829d6edfe65317)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-17 14:43:31 +01:00
Richard Purdie 654eadfa30 bitbake: bitbake: Update logger.warn() -> logger.warning()
python deprecated logger.warn() in favour of logger.warning(). This is only
used in bitbake code so we may as well just translate everything to avoid
warnings under python 3. Its safe for python 2.7.

(Bitbake rev: 676a5f592e8507e81b8f748d58acfea7572f8796)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-11 10:34:30 +01:00
Richard Purdie af5f423887 bitbake: siggen: Ensure tainted stamps are accounted for with writing custom stamps
sstate.bbclass for example writes siginfo files to a separate location
but we need to read taint data from the standard path.

(Bitbake rev: da444c9761ee15a59ea8880e3f812a5d3f1a1aaa)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-03 22:50:17 +01:00
Richard Purdie 47e9e125d7 bitbake: siggen: Fix nostamp taint handling
The taint values need to be passed from the server to the workers to
ensure they see the same stamp values. Also ensure that the "nostamp:"
prefix isn't included in the checksum value to match the server
calculation. This ensures the checksums are all consistent.

(Bitbake rev: f80ba20e90f3746f7faee3e0ff7f249025fec8ee)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-03 22:50:17 +01:00
Richard Purdie 80336270f3 bitbake: siggen: Add checksum recalculation/checking code
In theory all the information to recalcuate the task signatures was written
into the siginfo/sigdata files. In reality, some of the information was
written into the filename.

Firstly this patch duplicates that info into the file itself just for easy
of use since its small.

Secondly, we abstract out the existing "calculate the checksum" code for
the taskhash, and add a function to calculate the bashhash based on the
informaiton within the file.

Finally, we call these functions when we're writing out the data to check
that the data we're writing is consistent. I've found a couple of places
it wasn't and its good to know about these in advance, rather than having
a siginfo/sigdata file which a given hash in its filename but a contents
which give a different result.

This should all combine to avoid a certain class of checksum bugs making
it into world, and identifying problems in advance.

(Bitbake rev: 0f50a18d7a0ea0d68edd8e5217e29111f4b1ea0b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-03 22:50:17 +01:00
Richard Purdie 3e1b5e0685 bitbake: siggen: Fix check calculation problem with file_checksums
When I enabled debugging of the checksum code, I found the value calculated
from siginfo/sigdata files for do_fetch tasks never matched. This was due
to an error in the way the data was being stored for these, it wasn't ordered
correctly. This patch fixes things so the checksums calculated from
siginfo/sigdata files is correct when file checksums are present.

(Bitbake rev: 046c1be7594fae2eec3d1f242ba3e9a85f1a1880)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-03 15:53:44 +01:00
Richard Purdie 39b637ccd3 bitbake: siggen: Drop misleading duplicate method
The real method is a few lines later, this one is incorrect and
just causing confusion. Remove it.

(Bitbake rev: a896f263300f069400eae533be0daf5dedf41c95)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-03 15:53:44 +01:00
Markus Lehtonen 4f0ab27c6a bitbake: SignatureGeneratorBasic: make checksum cache file configurable
Define a new bitbake configuration variable BB_HASH_CHECKSUM_CACHE_FILE
that can be used to define the cache file to use for file checksum
cache.

(Bitbake rev: a965b390d6240e279c190b92b17c0573e9bd604c)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:41:16 +00:00
Markus Lehtonen 5177b1e4ac bitbake: SignatureGenerator: add method for saving the file checksum cache
Extend the API in order to be able to write out the file checksum cache
onto disk. SignatureGeneratorBasic class now implements a method that
update the fetcher local files checksum cache with the task file
dependency checksums.

(Bitbake rev: ecdabd321d48fa367b89ebffc00aa525b6eaa95c)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:41:15 +00:00
Richard Purdie 1ad3595e8c bitbake: siggen: Change exception note into a warning
This makes no sense as just a note, its at least a warning and useful
to get an idea of which codepath is failing.

(Bitbake rev: 0194cf0da24dc72dab0612cd54aa5190e6cd92f2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:34 +00:00
Richard Purdie cdc57f6116 bitbake: siggen: Make it clear why nostamp tasks signatures don't match
If you run bitbake-diffsigs against two differing sigdata files from
nostamp tasks it shows no difference despite the differing checksum.

Change the code so this shows up as a nostamp 'taint' and at least
makes the issue clearer to the end user.

(Bitbake rev: 97679d18955dadaa34f9450564e44da99984d140)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-01 07:43:38 +01:00
Richard Purdie 9e9d05205a bitbake: siggen: Don't crash if number of task dependencies change
If the number of task dependencies change you currently get
a traceback when using diffsigs.

(Bitbake rev: c6798b431571aae18bb8699ac6e3ec75b731d719)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-05-26 10:33:06 +01:00
Richard Purdie 745cebd635 bitbake: siggen: Ensure taskdata default functions exist in base class
The get/set_taskdata functions are now part of the API of the class,
ensure they exist in the base class definition so the noop handler
works.

[YOCTO #7233]

(Bitbake rev: 9b5b1bd7d77e3f5886f6c557d3b750de1f6d6025)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-01-23 14:39:44 +00:00
Richard Purdie 97d44bf526 bitbake: cache/fetch2/siggen: Ensure we track include history for file checksums
Currently, if you reference a file url, its checksum is included in the
task hash, however if you change to a different file at a different
location, perhaps taking advantage of the FILESPATH functionality, the
system will not reparse the file in question and change its checksum to
match the new file.

To correctly handle this, the system not only needs to know if the
existing file still exists or not, but also check the existance
of every file it would have looked at when computing the original file.

We already do this in the bitbake parsing code for class inclusion. This
change uses the same technique to log the file list we looked at and
if files in these locations exist when they previously did not, to
invalidate and reparse the file.

Since data stored in the cache is flattened text, we have to use a string
form of the data and split on the ":" character which is ugly, but is
an internal detail we can improve later if a better method is found.

The cache version changes to trigger a reparse since the previous
cache data is now incompatible.

[YOCTO #7019]

(Bitbake rev: 6c0706a28d72c591f1b75b6e3f3b645859387c7e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-01-08 09:52:52 +00:00
Paul Eggleton 7b70d5e8a4 bitbake: siggen: ensure nostamp tasks force dependent tasks to re-execute
If a nostamp task is depended on by a non-nostamp task, then we want the
signature of that task to change such that it re-executes afterwards.
This is an unusual situation, but we want this to work in OE in
externalsrc.bbclass so that compilation happens every time it is
requested.

(Bitbake rev: 73498afc3d45beede5b8f24a9acd523a1663b793)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-12-23 10:18:20 +00:00
Richard Purdie 01a61c8e6e bitbake: siggen: Fix shared work checksum mismatch/rebuild issues
Similar to the last shared work task signature bug, we've found another
one. Looking at the improved output of diffsigs in this case:

runtaskdeps changed from [
'autoconf_2.69.bb.do_populate_sysroot:virtual:native',
'gnu-config_20120814.bb.do_populate_sysroot:virtual:native',
'libgcc-initial_4.9.bb.do_patch:virtual:nativesdk'
] to [
'autoconf_2.69.bb.do_populate_sysroot:virtual:native',
'gcc-crosssdk-initial_4.9.bb.do_patch',
'gnu-config_20120814.bb.do_populate_sysroot:virtual:native'
]

so we can get a different task hash since libgcc sorts before gnu-config
and gcc sorts after it. We could do with a way of fixing this, the best
I can come up with is to include a single parent directory. Since
recipes are never at the top of any metadata trees I've seen, this
should suffice for now.

I'm planning to burn the concept of shared work within bitbake
and do something at the metadata level in the 1.8 timeframe as its just
too fragile as things stand and hard to fix well.

(Bitbake rev: d753644c67d163f338f2bdc3d600203e8b1a5734)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-11-08 11:14:31 +00:00
Richard Purdie 2db07ca8a3 bitbake: siggen: Add computed hash code to verify hash computation to dumpsig
This is useful code to double check the computed checksum value if nothing
else. Might as well have it in tree.

(Bitbake rev: 54ecf96c6f031927ee2410f6efde4e16f19bbf66)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-11-08 11:14:31 +00:00
Richard Purdie 636dcb84fe bitbake: siggen: Ensure we output if the ordering of runtaskdeps changes
Order of runtaskdeps is important. If the hashes differ we should print output.
This is complicated by shared work where the filenames themselves can differ,
but the checksum should not.

This fixes a case where two different checksums could show no output with
bitbake-diffsigs.

(Bitbake rev: 40c95cb9def282dc88234cd72ff462d7a01e47c1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-11-08 11:14:31 +00:00
Richard Purdie bb54fd0570 bitbake: siggen/runqueue/bitbake-worker: Improve siggen data transfer interface
We need to transfer some of the siggen data from the core/cooker into
the worker instances. There was a partial API created for this but
its ugly and its not possible to extend it from the siggen class.

This patch completes the interface/abstraction for the data and
means the class can extend/customise it in any siggen class.

(Bitbake rev: cf2d642052979d236185c5b8ca2c5478c06e62ae)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-09-11 17:31:17 +01:00
Richard Purdie 5a3a7e0f74 bitbake: siggen: Fix a subtle bug in hash calculation for shared work tasks
With a shared work task like gcc, the task can be run from a variety of
different recipes which may have different virtual extensions in place.

Depending on whether gcc-runtime or nativesdk-gcc-runtime's do_preconfigure
task is called for example will change the sorting of the task hashes due
to the way clean_basename currently works.

The correct thing to do here is sort on the base filename first, then any
extension when ordering the hashes. This means we do account for things
like recipes with both a native and non-native dependency but we also fix
the shared work case where we don't care whether it was a virtual version
or not.

(Bitbake rev: 2e80b5d10a5037ed6f0bc227a1f9b42529c87086)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-08-23 09:31:42 +01:00
Richard Purdie 60c40e5db5 bitbake: siggen: Print warning about tainted tasks
The big warning printed when people use -f is easily ignored/forgotten.
To raise user awareness, print a warning any time we include a tainted
stamp file into a build instead.

(Bitbake rev: 18f9bcbad059608e22fca20309314e1c399acec7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-06-01 14:29:32 +01:00
Richard Purdie 172095e09f bitbake: runqueue/siggen: Pass in commandline options to dump_sigs()
This allows the commandline options to be processed in the dump signature
code.

(Bitbake rev: ef8537a2e9b48f4fe065a165c102935aee2c9029)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-03-27 09:42:08 +00:00
Robert Yang d955d6a8f5 bitbake: siggen.py: fix the SignatureGenerator()
If we set:

BB_SIGNATURE_HANDLER = "noop"

Then we would get the following errors:

[snip]
File "runqueue.py", line 876, in RunQueue._start_worker(fakeroot=False, rqexec=None):
                "fakerootnoenv" : self.rqdata.dataCache.fakerootnoenv,
   >            "hashes" : bb.parse.siggen.taskhash,
                "hash_deps" : bb.parse.siggen.runtaskdeps,
AttributeError: 'SignatureGenerator' object has no attribute 'taskhash'
[snip]

This patch fixes the problem.

[YOCTO #5741]

(Bitbake rev: 2bfcb751891cf3b4050e996b3c8e28678c3a8bf4)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-01-28 00:52:57 +00:00
Martin Jansa 5e5a7fc37c bitbake: SignatureGenerator: Add empty implementation for dump_sigs
* bitbake -S throws exception when 'noop' signature handler is used

[YOCTO #5738]

(Bitbake rev: 53352e8d388b7fc4da73f95b93dcc087e76d0426)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-01-18 23:29:07 +00:00
Richard Purdie 19c75e92a1 bitbake: siggen: Remove fsync that is causing performance issues
This fsync was added for belt and braces protection for things like sstate
on NFS. To be honest, it probably doesn't buy much, if the rename isn't
atomic, all bets are off anyway and there are bigger issues to worry about.

The issue is that at the end of every task, the dump_sig() code is triggered
to save out information about the task and this was triggering an fsync(fd).

Whilst it may select the file descriptor, on file systems like ext4, it will
require large parts of the journal to be written out so it can have
significant impact. latencytop showed an average fsync() call overhead of about
2s and if that happens for 5000 tasks, the time mounts up. This blocks the next
task execution by that time.

We therefore drop the fsync since in reality its causing problems and is unlikely
to buy much.

(Bitbake rev: 46fd841319479f6079d850b3813e64bd8c2680a3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-01-18 23:29:07 +00:00
Richard Purdie 0b5476a3fa bitbake: siggen: Fix reversed difference output
The output when comparing siginfo files for dict_diff is reversed and shows
additions when things were removed and vice versa. This patch reverses the operation
so the changes are shown correctly and makes the output less confusing.

(Bitbake rev: 9b4142df36619099670740a5d3bc94e404ab2b56)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-12-20 12:26:06 +00:00
Richard Purdie a4457cf939 bitbake: siggen: When printing signatures recursively, limit the output
Currently the code prints all differences. If the task dependencies have changed hash,
we recurse into those and print those differences as well. This leads to a lot
of output. The reality is if the parents changed signature, we might as well just
say that and recurse with no other output since we're much more interested in how
the parents changed in nearly all cases. The changes in the parent are probably
the same ones we'd have printed at each level anyway.

By doing this we focus the output more carefully on the thing the user wants/needs
to see.

(Bitbake rev: 7a17fd6e51a76d3582c357b79f5ef86e1969650c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-12-20 12:26:05 +00:00
Paul Eggleton e9662e8926 bitbake: siggen: handle recipe path changes in siginfo files
Avoid storing paths to files in SRC_URI when writing out the the
file checksums to siginfo files. This prevents a move of the source
directory being reported by bitbake-diffsigs as files being removed and
then added (the signature itself is not affected since the file paths
have never been included in the signature).

This has required the format of the file checksums in the siginfo file
to be changed from a dict to a list of tuples (in order to handle
multiple files with the same name under different paths, which is
uncommon but possible); the code remains backwards-compatible with older
siginfo files that use a dict however.

Fixes [YOCTO #5245].

(Bitbake rev: e4d3077c5b0cc57964640512f3646c2d73c1d855)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-10-14 16:59:12 +01:00
Richard Purdie c971868360 bitbake: siggen: Use lookup cache exclusively
All the values we need are already guaranteed to be in the lookupcache
so rather than fetch variables again, just use the cache. This gives a
small performance improvement and simplifies the code.

(Bitbake rev: 8ffaba61da7f195d7c3b64dce35b6a56272aecae)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-09-17 14:11:03 +01:00
Richard Purdie d0f0e5d9e6 bitbake: runqueue: Split runqueue to use bitbake-worker
This is a pretty fundamental change to the way bitbake operates. It
splits out the task execution part of runqueue into a completely
separately exec'd process called bitbake-worker.

This means that the separate process has to build its own datastore and
that configuration needs to be passed from the cooker over to the
bitbake worker process.

Known issues:

* Hob is broken with this patch since it writes to the configuration
  and that configuration isn't preserved in bitbake-worker.
* We create a worker for setscene, then a new worker for the main task
  execution. This is wasteful but shouldn't be hard to fix.
* We probably send too much data over to bitbake-worker, need to
  see if we can streamline it.

These are issues which will be followed up in subsequent patches.

This patch sets the groundwork for the removal of the double bitbake
execution for psuedo which will be in a follow on patch.

(Bitbake rev: b2e26f1db28d74f2dd9df8ab4ed3b472503b9a5c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-06-14 12:52:56 +01:00
Richard Purdie 3e86a6cdde bitbake: lib: Use modern expcetion syntax
(Bitbake rev: a4a37b6a83faa62f61433122c4583e93e64f7372)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-05-09 22:28:24 +01:00
Richard Purdie 9b913576ea bitbake: siggen: Add missing error message parameter
There is clearly a missing parameter from this error message, add it.

(Bitbake rev: d98be5ea69b99fb7934fc3093422f139bc37a1cc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-05-09 22:28:04 +01:00
Martin Jansa 7ed4523963 bitbake: siggen: add quotes around variable values to see whitespace
* without this change it's hard to see what's changed:
  bitbake-diffsigs sstate-cache/*/sstate-depmodwrapper-cross-cortexa8-vfp-neon-oe-linux-gnueabi-1.0-r0-cortexa8-vfp-neon-*_populate-sysroot.tgz.siginfo
  basehash changed from 62f28c13172d38af917716d6d4b15ac5 to 9cf38842bcc79e0a4356856a64141548
  Variable TUNE_CCARGS value changed from  -march=armv7-a    -mthumb -mthumb-interwork -mfloat-abi=softfp -mfpu=neon -mtune=cortex-a8  to  -march=armv7-a    -mthumb -mthumb-interwork -mfloat-abi=softfp -mfpu=neon -mtune=cortex-a8
  Hash for dependent task depmodwrapper-cross_1.0.bb.do_install changed from 4494cb6c117309a9b4c541ad86c0aac5 to 1e3bcf2d14e5bd30a38350b41bb98db3

* with this change, we can see extra trailing space in first version:
  basehash changed from 62f28c13172d38af917716d6d4b15ac5 to 9cf38842bcc79e0a4356856a64141548
  Variable TUNE_CCARGS value changed from ' -march=armv7-a    -mthumb -mthumb-interwork -mfloat-abi=softfp -mfpu=neon -mtune=cortex-a8 ' to ' -march=armv7-a    -mthumb -mthumb-interwork -mfloat-abi=softfp -mfpu=neon -mtune=cortex-a8'
  Hash for dependent task depmodwrapper-cross_1.0.bb.do_install changed from 4494cb6c117309a9b4c541ad86c0aac5 to 1e3bcf2d14e5bd30a38350b41bb98db3

(Bitbake rev: 7d85b1a383feaa072503d71d3baeb3bdcd1012a8)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-03-04 11:38:30 +00:00