Use the TaskArtifacts event to scan the SDK and ESDK manifests
to cleanly collect the respective artifact files.
The previous method was broken when the SDK file deployment moved
from the do_populate_sdk[_ext] tasks to their sstate tasks. That
method is disabled (but not yet removed) in preparation for the
rest of refactor work for the parent #10283 work.
[YOCTO #10850]
(Bitbake rev: 1360d7b847cc01031edb2f4b289fac9560d72fa7)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the move over to argparse we've made the two signature file arguments
optional and thus if -t is not in use we need to explicitly check if at
least one signature file has been specified - and if not, show an error
and the command-line help.
(Bitbake rev: 9011366689b26305281fcb2d412dcacece917e18)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix a missing close quote in the layer add button, and
use the layer index for the xhr_layer call.
[YOCTO #11213]
(Bitbake rev: 54d61e7c64c97799de85b3563119f0f5d051675e)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A syntax fix inadvertantly broke the Toaster filter feature.
[YOCTO #11317]
(Bitbake rev: 29c9fa31d0f03553e24391568d2a5cb588595420)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the output is a TTY, add colour to the output in order to make it
easier to read. At the moment this is fairly basic, just add colour to
the "titles" of each change and to the diff output.
I tried to introduce this without changing the code too much - rather
than moving everything over to the new python formatting style, I've
introduced a color_format() function which takes care of the colour
formatting, either accepting additional format arguments or
alternatively leaving the caller to use the old-style formatting (%) to
insert values.
(Bitbake rev: 04a023c8fdea1e1812fcdcaf00345aab59f9abe1)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a variable value has changed and either the new or old value contains
spaces, a word diff should be appropriate and may be a bit more readable.
Import the "simplediff" module and use it to show a word diff (in the
style of GNU wdiff and git diff --word-diff).
Also use a similar style diff to show changes in the runtaskhashes list.
I didn't use an actual word-diff here since it's a little different - we
can be sure that the list is a list and not simply a free-format string.
(Bitbake rev: 20db6b6553c80e18afc4f43dc2495435f7477822)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we just want to drill down to the actual differences then we don't
need to see certain things in the output, e.g. basehash changing or the
signature of dependent tasks. This will be used for comparing signatures
within buildhistory-diff in OE-Core; the default mode as used by
bitbake-diffsigs and bitbake -S printdiff remains unchanged for the
moment.
(Bitbake rev: 6543a59b1ebd3194a7c6421cffc66ebe31a67c62)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the -t option which recurses to find the ultimate cause of a
signature change, it was hardcoded to take the last two executions of
the specified task. On the other hand, if you have two specific task
hashes (say from bitbake output, or some other tool) then you'll want to
pick those, so provide an option to specify those as well. (Note, the
new -s option needs to be specified alongside -t rather than instead of
it.)
(Bitbake rev: d9813b1a4223cf8dc80cab90e467ddf4bf8d1078)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Argparse is a bit easier to deal with than optparse, and since we're
about to add some options, migrate this script over.
(Bitbake rev: 7f130e0b5ce6cfc6b35176465f260092cd3b3d64)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the runtaskdeps list hasn't actually changed (but the signatures of
some of the tasks did) then it doesn't make sense to print out the old
and new lists as they are both the same and may be very long, e.g. for
do_rootfs in OE.
(Bitbake rev: cb170543605288b3e8badfac3a54c588f4c95413)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When dumping changes to signatures e.g. output of bitbake -s printdiff,
if for example a function has changed, it's much more readable to see a
unified diff of the changes rather than just printing the old function
followed by the new function, so use difflib to do that.
Note: I elected to keep to one item in the returned list per change,
rather than one line per line of output, so that the caller can still
look at changes individually if needed. Thus I've added some handling to
bitbake-diffsigs to split the change into lines so that each line is
displayed indented.
(Bitbake rev: 4d254ae63a35231c98e3f73f669b040ed1144042)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This logic doesn't work in practice, certainly not with current versions
where sigdata files are preserved in the stamps directory and therefore
there will often be multiple sigdata files - you can now easily get
files for the same signature from sstate and the stamps directory with the
result that bitbake-diffsigs reports nothing has changed. Instead, let's
change the find_siginfo function in OE-Core to simply not return
duplicates so we don't have to filter them out here.
(Bitbake rev: f0d7ab259d8ef95643e7229474b7850608aa4426)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If just one of the two signatures we want to compare aren't available,
report that one rather than misleadingly claiming both are missing.
(Bitbake rev: c87764b9147792a10efad3ed5378f36f0a055bc6)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Printing "pbzip2pbzip2_1.1.13.bb" is ugly, we need to add a separating
slash so that we get "pbzip2/pbzip2_1.1.13.bb" instead.
(Bitbake rev: 55cd4045a37afc954f4d5091f524756b266064fc)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We weren't picking the right files to compare here - according to the
order in which the list is sorted (by mtime), we need to be taking the
last two items and not the first two.
(Bitbake rev: 99f49b56115b1f8d1a0a0b911da62ffd1f997b5f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Valid machine names cannot include spaces anywhere
in the name and doing so will result in a build failure.
This implements a mechanism to alert the user against
such a misconfiguration and does not allow input of
such machine names.
[YOCTO #8721]
(Bitbake rev: 6fb642935a3787659aa316ca906025d2d87964cf)
Signed-off-by: Awais Belal <awais_belal@mentor.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if you specify a recrdeptask, it must exist for all recipes or
you get a python traceback. This is a bug and it should be possible to have
recipes which don't have the specified task.
As well as preventing such a traceback (which shouldn't happen, it should be
a user readable error), this allows us to fix issues in OE-Core which would
otherwise trigger the traceback.
(Bitbake rev: f93a77f2f188e18de9e3d812e86d77c2f3c71889)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This has long since been deprecated and is no longer used anywhere, FILESPATH
is the commonly used varaible which offers much more flexibility. Remove
the FILESDIR code and references from bitbake.
(Bitbake rev: 751c9dc51fd01fa64a1ff37ba2638110335f71af)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a few comments at the top of the file explaining what it's for, and
a comment pointing out that you need to increment the cache version when
changing any code that changes the output.
(Bitbake rev: c1780bc1872bb35bc28c76eff9110717f02d9db2)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add some tests to verify that we are extracting "contains" information
from python expressions in the code in the bb.data and bb.codeparser
modules.
(Bitbake rev: 88fda492df875dd79b7aecf1f34b38517fc1eb33)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ensure we handle bb.utils.contains_any() as separate items, rather than
how we handle contains() where every item must be in the list.
Additionally, enable handling bb.utils.filter() which for the purposes
of looking at dependencies is the same as contains_any().
Additionally bump the codeparser cache and recipe cache versions to
invalidate the user's existing caches (ensuring that the changes take
effect and avoiding "taskhash mismatch" errors respectively).
(Bitbake rev: 496e3c84820a2a889d99d3604659e47a550941d5)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code that determines variable dependencies uses the codeparser to
find references to "contains" type operations e.g. bb.utils.contains().
That function can take multiple items to check, and all specified items
have to be present. However this code didn't handle that - it assumed
that only one item would be specified and thus it was treating the
multiple items as a single item with spaces in between. Split the value
and check if all words are present in order to determine whether the
check is "set" or "unset".
(Bitbake rev: 272f1245acdd4fb85cb78612aa03627a9c246d8f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes [YOCTO #9962]
The new output for -D was copied into the manual and merged by
the engineering team before checking to see if the manual
would make properly. The next output for -D introduced an
error through the string "<task>". The angled bracket
characters are illegal in docbook and must be replaced by
"<" and ">", respectively. I made this fix.
(Bitbake rev: 0476ec4605c94b9e0208cfd3fa48ecf3e3f04181)
Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the fetcher retrieves file:// URLs, there is no lock file being
used. This means that in case two separate tasks (typically from two
concurrent invocations of bitbake) want to download the same file://
URL at the same time, there is a very small chance that they also end
up wanting to create a symbolic link to the file at the same time.
This would previously lead to one of the tasks failing as the other
task would have created the link.
(Bitbake rev: 58a03531c8183b165bb7dcad86d8559c92bc150d)
Signed-off-by: Peter Kjellerstedt <pkj@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Normally bb.utils.which() is used by the unpack code to find a file in a variety
of places, but it is useful as a slightly more powerful version of os.which().
Support this by allowing it to only return matches which are executable files,
instead of just the first filename that matches.
(Bitbake rev: c0b94f02f0cba7a424aaa16cf98c0f7a3f62b889)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
URL decoding was improved in the core a while ago and this looks like
a leftover from those times which caused urls needing a user/password to
fail. Use the parameters from the core instead of the broken split
implementation.
[YOCTO #11262]
(Bitbake rev: 6a917ec99d659e684b15fa8af94c325172676062)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Turn reasons from a list into a string (usually there will be only one
reason, but the interface provides for more than one) and state up
front that the recipe is unavailable for clarity
* Use quotes around invalid recipe name
(Bitbake rev: 8922f1d23400049d2894a97915a533769a24ca07)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This function calls cooker.findBestProvider() but didn't handle the fact
that that function returns a tuple (None, None, None, None) when there
is no matching recipe. (This fixes devtool in OpenEmbedded showing a
traceback when an invalid recipe is specified instead of a proper error
message.)
(Bitbake rev: 54a4757ca706afc6e98c7692f960592e80cab12b)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit looks to see if FETCHCMD_s3 is set and if not, sets
it.
This is needed because I've use cases where I don't use aws, but
s3cmd (due to license).
(Bitbake rev: fdeaed70a7d1ff8be1a1de937cb864130b0c2c86)
Signed-off-by: Elizabeth 'pidge' Flanagan <pidge@toganlabs.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BufferedLogger was sending log records to the target logger without
checking if the logger is enabled for the level of the record - and
handle() doesn't check this either (it's normally checked earlier when
the relevant log function is called e.g. logger.debug()), leading for
example to debug messages from codeparser getting printed when the log
level for the main BitBake logger was set to logging.WARNING.
(Bitbake rev: 968a77388dd1a24c1dadec6ce49bf53ebb5b643f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Imagine you have an sstate mirror accessed over http and an SSTATE_MIRRORS
which maps file:// urls to http:// urls.
File urls set donestampneeded = False, http urls don't. This can result in
races in the try_mirror_url() code since it will trigger new downloads after
aquiring the lockfile as verify_donestamp() doesn't look at origud and there
is no donestamp.
verify_donestamp() already has code to look at origud, we're just missing
some code at the start of the function to do this. Fix it to avoid
these races.
(Bitbake rev: b8b14d975a254444461ba857fc6fb8c725de8874)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Expanded and clarify documentation for the -v, --verbose and -D, --debug
options.
[YOCTO #9962]
(Bitbake rev: 921592026c69287cdb40ffd90944d5944f28e2c3)
Signed-off-by: Diana Thayer <garbados@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you had a script that uses tinfoil and it failed to connect to the
BitBake server, you did't see any of the expected messages - this was
because client-side logging wasn't being handled at all. Since you'll
almost always want this when using tinfoil, have it use the new
bb.msg.logger_create() function to enable client-side logging by
default.
Relates to [YOCTO #11185].
(Bitbake rev: 824e73e0f3eaa96b4d84da7e31f9a17ce5c5d7ee)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We use this code to set up a logger with colour in a number of different
places, so create one function that does this and make some of bitbake's
utility scripts use it.
(Bitbake rev: b1ba7d1cc8ec33e2d081230287abd07f52136097)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We call git ls-remote to get the latest revision from a git repository,
however by calling runfetchcmd() we can end up recursively running
git ls-remote a number of times with OE e.g. if ${SRCPV} is in PV, ${PV}
is in WORKDIR, and ${WORKDIR} is in PATH (as a result of recipe-specific
sysroots), our call to runfetchcmd() exports PATH so _lsremote() will
get called again - with the end result that we run git ls-remote 30
times in quick succession (!). Prevent that from happening by using a
guard variable and returning a dummy value if it's called recursively.
Fixes [YOCTO #11185].
(Bitbake rev: ff1ccd1db5d70b3fc9ad0d3e8f3d7b804c22bf36)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently shows the name of the signature files that were found when
--task is used.
(Bitbake rev: 751b06c25d22eea8240f9429cb49874082245e52)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently shows the name of the signature file that was found when
--task is used.
(Bitbake rev: 84533c6dbd175a51f4dd59735161adfd10056888)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use --task <recipe> <task> to dump the signature info for a given
recipe and task. This is similar to the --task option of
bitbake-diffsigs.
(Bitbake rev: bdc4356c7afc542b67b78e4e5225b813d7668ecd)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The dependency graph in the bb.event.DepTreeGenerated and the
corresponding pn-buildlist and task-depends.dot from "bitbake -g"
contained entries like multiconfig:qemuarm.gcc (dot as last separator)
instead of the correct multiconfig:qemuarm:gcc (colon as separator).
(Bitbake rev: cccd1578d84f041cd0c2dcddb91f317c69af70de)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Displaying the environment data in a sorted manner makes it easier
to compare data between different setups and means you can know
where to find specific entries.
(Bitbake rev: baa417e44b009149eb1dfb07d5a488740b6e68ef)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
FetchError isn't defined, use bb.fetch2.FetchError in this context.
(Bitbake rev: 945fa980e027753df2c21d84eb63dcaddb2caaee)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The touch of .done explicitly specifies the path, so there's no need for
workdir=, and "os.path.join('.')" is identical to just '.'.
(Bitbake rev: 955cbfdaa2400d15ec428b65848e6835c9f44860)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
list.index() isn't a particularly efficient operation, so keep track of our
position via enumerate() instead, which is more pythonic as well.
(Bitbake rev: dec6e90a4d27ee335e9c78aeebd277098fec94d1)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Various run.do_xxx files export dozens of variables.
Presently they are in random order. Among other things,
this makes it difficult to compare two files for relevant changes.
This patch ensures they are enumerated/printed in a sorted
(alphabetical) order.
(Bitbake rev: 27b501d1400d1942b5ba9f35218ca7aacd9bfefe)
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
David Reyna
Paul Eggleton
Markus Lehtonen
Belal, Awais
Richard Purdie
Scott Rifenbark
Peter Kjellerstedt
Ross Burton
Elizabeth 'pidge' Flanagan
Diana Thayer
Patrick Ohly
Christopher Larson
Juro Bystricky
Joshua Lock