Argparse is a bit easier to deal with than optparse, and since we're
about to add some options, migrate this script over.
(Bitbake rev: 7f130e0b5ce6cfc6b35176465f260092cd3b3d64)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the runtaskdeps list hasn't actually changed (but the signatures of
some of the tasks did) then it doesn't make sense to print out the old
and new lists as they are both the same and may be very long, e.g. for
do_rootfs in OE.
(Bitbake rev: cb170543605288b3e8badfac3a54c588f4c95413)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When dumping changes to signatures e.g. output of bitbake -s printdiff,
if for example a function has changed, it's much more readable to see a
unified diff of the changes rather than just printing the old function
followed by the new function, so use difflib to do that.
Note: I elected to keep to one item in the returned list per change,
rather than one line per line of output, so that the caller can still
look at changes individually if needed. Thus I've added some handling to
bitbake-diffsigs to split the change into lines so that each line is
displayed indented.
(Bitbake rev: 4d254ae63a35231c98e3f73f669b040ed1144042)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This logic doesn't work in practice, certainly not with current versions
where sigdata files are preserved in the stamps directory and therefore
there will often be multiple sigdata files - you can now easily get
files for the same signature from sstate and the stamps directory with the
result that bitbake-diffsigs reports nothing has changed. Instead, let's
change the find_siginfo function in OE-Core to simply not return
duplicates so we don't have to filter them out here.
(Bitbake rev: f0d7ab259d8ef95643e7229474b7850608aa4426)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If just one of the two signatures we want to compare aren't available,
report that one rather than misleadingly claiming both are missing.
(Bitbake rev: c87764b9147792a10efad3ed5378f36f0a055bc6)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Printing "pbzip2pbzip2_1.1.13.bb" is ugly, we need to add a separating
slash so that we get "pbzip2/pbzip2_1.1.13.bb" instead.
(Bitbake rev: 55cd4045a37afc954f4d5091f524756b266064fc)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We weren't picking the right files to compare here - according to the
order in which the list is sorted (by mtime), we need to be taking the
last two items and not the first two.
(Bitbake rev: 99f49b56115b1f8d1a0a0b911da62ffd1f997b5f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Valid machine names cannot include spaces anywhere
in the name and doing so will result in a build failure.
This implements a mechanism to alert the user against
such a misconfiguration and does not allow input of
such machine names.
[YOCTO #8721]
(Bitbake rev: 6fb642935a3787659aa316ca906025d2d87964cf)
Signed-off-by: Awais Belal <awais_belal@mentor.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if you specify a recrdeptask, it must exist for all recipes or
you get a python traceback. This is a bug and it should be possible to have
recipes which don't have the specified task.
As well as preventing such a traceback (which shouldn't happen, it should be
a user readable error), this allows us to fix issues in OE-Core which would
otherwise trigger the traceback.
(Bitbake rev: f93a77f2f188e18de9e3d812e86d77c2f3c71889)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This has long since been deprecated and is no longer used anywhere, FILESPATH
is the commonly used varaible which offers much more flexibility. Remove
the FILESDIR code and references from bitbake.
(Bitbake rev: 751c9dc51fd01fa64a1ff37ba2638110335f71af)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a few comments at the top of the file explaining what it's for, and
a comment pointing out that you need to increment the cache version when
changing any code that changes the output.
(Bitbake rev: c1780bc1872bb35bc28c76eff9110717f02d9db2)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add some tests to verify that we are extracting "contains" information
from python expressions in the code in the bb.data and bb.codeparser
modules.
(Bitbake rev: 88fda492df875dd79b7aecf1f34b38517fc1eb33)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ensure we handle bb.utils.contains_any() as separate items, rather than
how we handle contains() where every item must be in the list.
Additionally, enable handling bb.utils.filter() which for the purposes
of looking at dependencies is the same as contains_any().
Additionally bump the codeparser cache and recipe cache versions to
invalidate the user's existing caches (ensuring that the changes take
effect and avoiding "taskhash mismatch" errors respectively).
(Bitbake rev: 496e3c84820a2a889d99d3604659e47a550941d5)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code that determines variable dependencies uses the codeparser to
find references to "contains" type operations e.g. bb.utils.contains().
That function can take multiple items to check, and all specified items
have to be present. However this code didn't handle that - it assumed
that only one item would be specified and thus it was treating the
multiple items as a single item with spaces in between. Split the value
and check if all words are present in order to determine whether the
check is "set" or "unset".
(Bitbake rev: 272f1245acdd4fb85cb78612aa03627a9c246d8f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes [YOCTO #9962]
The new output for -D was copied into the manual and merged by
the engineering team before checking to see if the manual
would make properly. The next output for -D introduced an
error through the string "<task>". The angled bracket
characters are illegal in docbook and must be replaced by
"<" and ">", respectively. I made this fix.
(Bitbake rev: 0476ec4605c94b9e0208cfd3fa48ecf3e3f04181)
Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the fetcher retrieves file:// URLs, there is no lock file being
used. This means that in case two separate tasks (typically from two
concurrent invocations of bitbake) want to download the same file://
URL at the same time, there is a very small chance that they also end
up wanting to create a symbolic link to the file at the same time.
This would previously lead to one of the tasks failing as the other
task would have created the link.
(Bitbake rev: 58a03531c8183b165bb7dcad86d8559c92bc150d)
Signed-off-by: Peter Kjellerstedt <pkj@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Normally bb.utils.which() is used by the unpack code to find a file in a variety
of places, but it is useful as a slightly more powerful version of os.which().
Support this by allowing it to only return matches which are executable files,
instead of just the first filename that matches.
(Bitbake rev: c0b94f02f0cba7a424aaa16cf98c0f7a3f62b889)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
URL decoding was improved in the core a while ago and this looks like
a leftover from those times which caused urls needing a user/password to
fail. Use the parameters from the core instead of the broken split
implementation.
[YOCTO #11262]
(Bitbake rev: 6a917ec99d659e684b15fa8af94c325172676062)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Turn reasons from a list into a string (usually there will be only one
reason, but the interface provides for more than one) and state up
front that the recipe is unavailable for clarity
* Use quotes around invalid recipe name
(Bitbake rev: 8922f1d23400049d2894a97915a533769a24ca07)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This function calls cooker.findBestProvider() but didn't handle the fact
that that function returns a tuple (None, None, None, None) when there
is no matching recipe. (This fixes devtool in OpenEmbedded showing a
traceback when an invalid recipe is specified instead of a proper error
message.)
(Bitbake rev: 54a4757ca706afc6e98c7692f960592e80cab12b)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit looks to see if FETCHCMD_s3 is set and if not, sets
it.
This is needed because I've use cases where I don't use aws, but
s3cmd (due to license).
(Bitbake rev: fdeaed70a7d1ff8be1a1de937cb864130b0c2c86)
Signed-off-by: Elizabeth 'pidge' Flanagan <pidge@toganlabs.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BufferedLogger was sending log records to the target logger without
checking if the logger is enabled for the level of the record - and
handle() doesn't check this either (it's normally checked earlier when
the relevant log function is called e.g. logger.debug()), leading for
example to debug messages from codeparser getting printed when the log
level for the main BitBake logger was set to logging.WARNING.
(Bitbake rev: 968a77388dd1a24c1dadec6ce49bf53ebb5b643f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Imagine you have an sstate mirror accessed over http and an SSTATE_MIRRORS
which maps file:// urls to http:// urls.
File urls set donestampneeded = False, http urls don't. This can result in
races in the try_mirror_url() code since it will trigger new downloads after
aquiring the lockfile as verify_donestamp() doesn't look at origud and there
is no donestamp.
verify_donestamp() already has code to look at origud, we're just missing
some code at the start of the function to do this. Fix it to avoid
these races.
(Bitbake rev: b8b14d975a254444461ba857fc6fb8c725de8874)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Expanded and clarify documentation for the -v, --verbose and -D, --debug
options.
[YOCTO #9962]
(Bitbake rev: 921592026c69287cdb40ffd90944d5944f28e2c3)
Signed-off-by: Diana Thayer <garbados@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you had a script that uses tinfoil and it failed to connect to the
BitBake server, you did't see any of the expected messages - this was
because client-side logging wasn't being handled at all. Since you'll
almost always want this when using tinfoil, have it use the new
bb.msg.logger_create() function to enable client-side logging by
default.
Relates to [YOCTO #11185].
(Bitbake rev: 824e73e0f3eaa96b4d84da7e31f9a17ce5c5d7ee)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We use this code to set up a logger with colour in a number of different
places, so create one function that does this and make some of bitbake's
utility scripts use it.
(Bitbake rev: b1ba7d1cc8ec33e2d081230287abd07f52136097)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We call git ls-remote to get the latest revision from a git repository,
however by calling runfetchcmd() we can end up recursively running
git ls-remote a number of times with OE e.g. if ${SRCPV} is in PV, ${PV}
is in WORKDIR, and ${WORKDIR} is in PATH (as a result of recipe-specific
sysroots), our call to runfetchcmd() exports PATH so _lsremote() will
get called again - with the end result that we run git ls-remote 30
times in quick succession (!). Prevent that from happening by using a
guard variable and returning a dummy value if it's called recursively.
Fixes [YOCTO #11185].
(Bitbake rev: ff1ccd1db5d70b3fc9ad0d3e8f3d7b804c22bf36)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently shows the name of the signature files that were found when
--task is used.
(Bitbake rev: 751b06c25d22eea8240f9429cb49874082245e52)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently shows the name of the signature file that was found when
--task is used.
(Bitbake rev: 84533c6dbd175a51f4dd59735161adfd10056888)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use --task <recipe> <task> to dump the signature info for a given
recipe and task. This is similar to the --task option of
bitbake-diffsigs.
(Bitbake rev: bdc4356c7afc542b67b78e4e5225b813d7668ecd)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The dependency graph in the bb.event.DepTreeGenerated and the
corresponding pn-buildlist and task-depends.dot from "bitbake -g"
contained entries like multiconfig:qemuarm.gcc (dot as last separator)
instead of the correct multiconfig:qemuarm:gcc (colon as separator).
(Bitbake rev: cccd1578d84f041cd0c2dcddb91f317c69af70de)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Displaying the environment data in a sorted manner makes it easier
to compare data between different setups and means you can know
where to find specific entries.
(Bitbake rev: baa417e44b009149eb1dfb07d5a488740b6e68ef)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
FetchError isn't defined, use bb.fetch2.FetchError in this context.
(Bitbake rev: 945fa980e027753df2c21d84eb63dcaddb2caaee)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The touch of .done explicitly specifies the path, so there's no need for
workdir=, and "os.path.join('.')" is identical to just '.'.
(Bitbake rev: 955cbfdaa2400d15ec428b65848e6835c9f44860)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
list.index() isn't a particularly efficient operation, so keep track of our
position via enumerate() instead, which is more pythonic as well.
(Bitbake rev: dec6e90a4d27ee335e9c78aeebd277098fec94d1)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Various run.do_xxx files export dozens of variables.
Presently they are in random order. Among other things,
this makes it difficult to compare two files for relevant changes.
This patch ensures they are enumerated/printed in a sorted
(alphabetical) order.
(Bitbake rev: 27b501d1400d1942b5ba9f35218ca7aacd9bfefe)
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is what was reported when variable expansion in the 'depends'
varflag of a task was broken:
ERROR: Error for .../refkit-image-common.bb, dependency ${@ does not contain exactly one ':' character.
Task 'depends' should be specified in the form 'packagename:task'
It's not clear which task had this broken 'depends' and while one can
guess that variable expansion failed, the full expression isn't
printed either.
This is more useful:
ERROR: Error for .../refkit-image-common.bb:do_stage_swupd_inputs[depends], dependency ${@ in ' virtual/fakeroot-native:do_populate_sysroot ${@ ' '.join(['bundle-refkit-image-common-%s:do_swupd_list_bundle' % x for x in '${SWUPD_BUNDLES}'.split()]) } ' does not contain exactly one ':' character.
Task 'depends' should be specified in the form 'packagename:task'
The 'depends' part gets repeated intentionally, to ensure that it doesn't get overlooked.
(Bitbake rev: eeeed7693710e4a78fec639b8050d8efe4ea3c33)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The data we read from an HTTPConnection comes in the form of bytes, but
we need it as a string, so in Python 3 we need to decode it (missed in
the Python 3 migration).
(Bitbake rev: 7e6a3cd1472e1a1c8304b46611e1676914a68b59)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Enable the following operations from a remote datastore to affect the
other end:
* setVarFlag()
* delVar()
* delVarFlag()
* renameVar()
In practice I don't expect these to be used much, but they should be
present so that the implementation is at least reasonably filled out
and that the tests pass.
Also add tests for the interface, mostly by subclassing the existing
local test classes so that they are using a remote datastore. (These
don't actually test remote usage via tinfoil, just that the
datastore's interface can be used.)
(Bitbake rev: 282dc0719d22a39df746eea762ebe05c66aa8f8a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was a huge gap in the remote datastore code introduced in the
tinfoil2 rework - we weren't handling overrides at all, since these are
stored separately from the actual data in the DataSmart object. Thus,
when a datastore actually represents a remote datastore we need to go
back to that remote datastore to get the override data as well, so
introduce code to do that.
To avoid a second round-trip I had to modify the _findVar() function to
return the override data as well. This will increase the overhead a
little when that data is superfluous, but without making the function
even uglier I don't think there's a way to avoid that.
(Bitbake rev: 4f9d6f060ed247fb6fa2f45668a892a1788d3f91)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In 2bc4d35fb32defc59cd6ed1fc87e35924c201a5c we fixed cases where an inactive
remove override was influecing the value of a variable. This adds a test
case for this issue.
(Bitbake rev: 1ff9b3c669fa187f152de7b8b57d14c2468d926c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the regular expression for matching a variable name was amended
with allowing the ~ character as part of the variable name, this was
never done to the regular expression that matches export
lines. Similarly, the regular expression that was used for matching
unset variables also used the one without support for the ~ character.
This unifies the regular expressions. For good measures it also
corrects the regular expression used to match a variable flag name for
the unset command to match the one used when setting a variable flag.
(Bitbake rev: acd2fd74ed467dc85ec75d5d0815f43e493f29bf)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For some reason, BBProgress.setextra() always adds a space at the end.
This prevents the task progress bars from filling the entire width of
the terminal, making them stop one character short. This looks odd
when shown together with the main progress bar, which does fill the
entire terminal width.
(Bitbake rev: 6540e98090d6a88607489a23c063be338bcc3b57)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For example:
$ bitbake core-image-minimal --runall unpack
...
NOTE: Tasks Summary: Attempted 326 tasks of which 88 didn't need to be rerun and all succeeded.
$ bitbake core-image-minimal --runall patch
NOTE: Tasks Summary: Attempted 453 tasks of which 332 didn't need to be rerun and all succeeded.
This can replace fetchall as well:
$ bitbake core-image-minimal --runall fetch
NOTE: Tasks Summary: Attempted 135 tasks of which 119 didn't need to be rerun and all succeeded.
(Bitbake rev: 7c0fa6ba66cdb956b37d94055307cde857633df9)
Signed-off-by: Matthew McClintock <msm-oss@mcclintock.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some task events are missing the 'virtual:native[sdk]:' prefixes.
The Toaster has code to help match missing prefixes, but needs
additional help resolving between 'native:' and 'nativesdk:', by
way of the '_package' event field.
[YOCTO #10849]
(Bitbake rev: e455e40ba309837903b9e2d5f1dff55cce1135de)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Treat tar files compressed with 7-Zip in the same way as tar files
compressed with other compression formats.
(Bitbake rev: 363a0f54dc7d9930537f0df25173fa31ca1f98ac)
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Class for fetching files from Amazon S3 using the AWS Command Line
Interface. The aws tool must be correctly installed and configured
prior to use.
The class supports both download() and checkstatus(), which therefore
allows S3 mirrors to be used for SSTATE_MIRRORS.
(Bitbake rev: 6fe07ed25457dd7952b60f4b2153d56b15d5eea6)
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Forcing the use of "\n" in mirror variables is pointless, we can just require that
there are pairs of values.
(Bitbake rev: 044fb04dbe69313ee6908bf4d3cee7f797d0c41c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes [YOCTO #10405]
The section on the SVN Fetcher was missing information on the
"path_spec" option. I added this option and also updated the
examples at the bottom of the section to include that parameter.
Also, made the other two examples consistent.
I also removed the "date" parameter.
Also, updated the "protocol" parameter as well as the "modify"
parameter. For "modify" I removed the reference to "rsh". I
applied a small wording change to the "protocol" parameter.
Finally, I added a new "ssh" parameter.
(Bitbake rev: 3ce6169afa646ef2b847e5fbabfe0191c93928b7)
Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The package-depends.dot and pn-depends.dot files are inaccurate,
missing out key dependencies such those made via the [depends]
flags. As such they can be misleading to the user.
They mainly exist for historical reasons, coming from a time
before we had task based execution.
This commit removes the two dated file formats and replaces
them with a recipe-depends.dot which is a flattened version
of task-depends.dot.
The old format files are removed if present so that the user
can't get confused about why data might not match between files.
The code is also rewritten to use 'with f: f.write()' syntax as
is more commonly used now. Also update the docs to match the change.
(Bitbake rev: f82537d27f2a5bf9d576aa841593db9ec0985ea8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Cleanup some more usage of bb.data APIs in the fetchers.
(Bitbake rev: 9752fd1c10b8fcc819822fa6eabc2c1050fcc03b)
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bb.utils.filter() function can be used to filter a variable
containing whitespace separated words based on another set of words.
It has been modeled after the bb.utils.contains_any() function.
A typical example of how it can be used is to simplify constructs for
PACKAGECONFIG that depend on DISTRO_FEATURES:
-PACKAGECONFIG ?= "\
- ${@bb.utils.contains('DISTRO_FEATURES', 'acl', 'acl', '', d)} \
- ${@bb.utils.contains('DISTRO_FEATURES', 'selinux', 'selinux', '', d)} \
-"
+PACKAGECONFIG ?= "${@bb.utils.filter('DISTRO_FEATURES', 'acl selinux', d)}"
(Bitbake rev: 03ae490366d2046f5b5c185fe4ec2adf1b0a902e)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Python3 regards b"" as False so it is not being converted to a string by
d0f904d407f57998419bd9c305ce53e5eaa36b24. This confusingly causes three
different potential types for each member of the returned tuple.
Let's just assume that everything that's not None is a bytes object and
convert it to a string.
(Bitbake rev: 0cf5589b7fb3582a6caca5014c4d8152347df545)
Signed-off-by: Mike Crowe <mac@mcrowe.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Most angular2 packages have names of the form @angular/xxx.
The / obviously can't be used in a file name, replace it with -.
(Bitbake rev: d3bd41d0ec9621307c362b394872b18b8b7ed8d6)
Signed-off-by: Anders Darander <anders@chargestorm.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A number of npm packages use @ as a leading chararacter.
Examples are most of the angular2 packages.
(Bitbake rev: 628c4bf6c89b3d62c9b864380b5c8e131a899bff)
Signed-off-by: Anders Darander <anders@chargestorm.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The functionality around the 'rsh' parameter seemed to be broken. The
'rsh' parameter was only used when 'protocol' was set to 'svn+ssh' which
is confusing. The 'rsh' parameter was used for setting the value of
'svn_RSH' environment variable, which however, is not supported by svn
(not at least according to SVN documentation).
This patch removes the 'rsh' parameter and replaces it with 'ssh'. This
new (optional) parameter is used when svn+ssh protocol is used and it
can be used to specify the ssh program used by svn. This is achieved by
setting the SVN_SSH environment variable which is mentioned in SVN
documentation.
(Bitbake rev: 5b364b02270b0d7c2b7ca8d67fa2731bf93720ee)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we removed the postinst sstate dependency handling code from
setscene_depvalid, we noticed things being installed into the old
style sysroot for rootfs tasks which should not have been there,
causing a performance regression.
Analysis revealed that setscene dependencies were "bubbling" over
sstate tasks when they should have been stopping there. The
'continue' added by this patch avoids this issue and eusures
sstate tasks remain contained to their specific chains.
There was another bug in the code this exposed where the acconting
for tasks as they were removed from sq_revdeps was not correct. In
fixing this, what looks like a workaround in another test can then
be simplified.
After this change, populate_sysroot tasks are no longer depending
on package_write_rpm tasks for example, which would make no sense.
A before/after analysis of image dependencies only revealed improved
dependencies after this change.
Recipe specific sysroots did highlight the issue here since the
behaviour of the sysroot dependencies (and processing with depvalid)
was not matching what bitbake itself was doing, with bitbake being
incorrect. Failures were 'safe' in that too many dependencies would
get installed.
(Bitbake rev: 5ef2cb50041fa7106c8de170af73d2a54cb0b1f0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The tinfoil.prepare method can raise Exceptions when is parsing initial
data so add this call inside try, finally to avoid get blocked for
don't call tinfoil.shutdown().
The tinfoil_init function was remove because isn't make sense now since
tinfoil.prepare() needs to be inside try, finally closures.
Example of raised exception and gets blocked:
$ bitbake-layers add-layer ~/repos/meta-openembedded/meta-python/
Traceback (most recent call last):
File "/home/alimon/repos/poky/bitbake/bin/bitbake-layers", line 83, in
main
tinfoil.prepare(True)
...
File "/home/alimon/repos/poky/bitbake/lib/bb/tinfoil.py", line 268, in
run_command
raise TinfoilCommandFailed(result[1])
bb.tinfoil.TinfoilCommandFailed: Traceback (most recent call last):
File "/home/alimon/repos/poky/bitbake/lib/bb/command.py", line 81, in
runCommand
result = command_method(self, commandline)
...
File "/home/alimon/repos/poky/bitbake/lib/bb/cooker.py", line 1314, in
handleCollections
raise CollectionError("Errors during parsing layer configuration")
bb.cooker.CollectionError: Errors during parsing layer configuration
(Bitbake rev: 61ebe9a026652e32f9482f66fc3fe7a8672fc093)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The runqueue expects to be able to run 'bitbake-worker' from PATH, but
for example in the OE extensible SDK, tinfoil is used within devtool
where this isn't the case (we're not exposing bitbake to the user, thus
bitbake's bin directory isn't in PATH).
This fixes devtool modify usage within the extensible SDK which has
apparently been broken since the tinfoil2 changes went in.
Fixes [YOCTO #11034].
(Bitbake rev: 52d87e4a58a52174e8e0b6297abaac8fd93ffcc3)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- In some cases the file descriptor
is held by nfs client and none of os.path.* is catching
that, it could mean that error is not doled out because
client has cached the stat info. In this case we are
out of luck. Needed to catch IOError, which would be
causing the Stale error.
- In download method, update_stamp is invoked
md5sum validation which is found to be throwing
Stale errors.
- Added error handling to fix the stale errors.
(Bitbake rev: 5a53e7d7b017769a6eb0f0a6335735a1fe51a5ec)
Signed-off-by: Balaji Punnuru <balaji_punnuru@cable.comcast.com>
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For spelling's sake, rename Python routine "setup_revisons" to
"setup_revisions."
(Bitbake rev: 4df59b027c02ef39d72476251ccd3fd62fc20bf6)
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 05f5421b2e44cd58c5912848de43d5884d070150)
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When constructing an older style single regex, it's possible for BBMASK
to end up beginning with '|', which matches and masks _everything_.
(Bitbake rev: 56ad67017e601c7e0f6085ca84e29c28d8d4519f)
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The package-depends.dot and pn-depends.dot files are inaccurate, missing
out key dependencies such those made via the [depends] flags. As such
they can be misleading to the user.
They mainly exist for historical reasons, coming from a time before
we had task based execution.
This commit removes the two dated file formats and replaces them with
a recipe-depends.dot which is a flattened version of task-depends.dot.
The old format files are removed if present so that the user can't get
confused about why data might not match between files.
The code is also rewritten to use 'with f: f.write()' syntax as is more
commonly used now.
Also update the docs to match the change.
(Bitbake rev: d3e182bc18ff2894f1efc8aad3d508dd432c996e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The depexp UI has been replaced with taskexp, also remove old
references to goggle and hob.
(Bitbake rev: 814c2b232ea5f88b5ad5031fb9d389c9c594aa01)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The data displayed by the depexp UI is buggy, bitbake can't convey
accurate runtime dependency information.
Convert the UI to become taskexp ("Task Explorer") which provides
a graphical way of exploring the task-depends.dot file. This data
is accurate and being able to visualise the data is an often requested
feature as standard dot file viewers can't usualy cope.
(Bitbake rev: 416a496edc692982af2feec768ac4df8fb799853)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Also run sanity check for all additional multiconfig configurations
as listed in BBMULTICONFIG, not just the "default" configuration.
[YOCTO#10810]
(Bitbake rev: b9777374cbe63cdcf80767033f628c26d23e90f9)
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add wic.bmap file extension support for toaster
(Bitbake rev: 6a3f6f1b5cc9ec69b28bebf70daed7579c60cd9e)
Signed-off-by: Sujith H <sujith.h@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was a superfluous section in toaster start/stop script that was
setting the TOASTER_DIR and had a set of comments around that setting.
This was done in two places and only the last one was effective. This
patch removes the spurious section to make it clearer what was
happening and what TOASTER_DIR was actually getting set to.
(Bitbake rev: e551e73a094f5a1937bce78de9e41908bf3e07c3)
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The toaster.sqlite database was located in TOASTER_DIR/build. This
meant that if you named your build directory something else (like cow),
Toaster would fail to make/find the database. TOASTER_DIR is on the
whitelist unlike BUILDDIR and we need to be able to write there anyway
given our current layout so this should not disrupt anything.
[YOCTO #9992]
(Bitbake rev: 2c48168a86309c0cf2be793e7409a78ba21fca14)
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was a superfluous section in toaster start/stop script that was
setting the TOASTER_DIR and had a set of comments around that setting.
This was done in two places and only the last one was effective. This
patch removes the spurious section to make it clearer what was
happening and what TOASTER_DIR was actually getting set to.
(Bitbake rev: 1f0eac0a172a4fbe1799675f2c3ce989743bd862)
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that the datastore works dynamically we don't need the update_data calls
so we can just remove them. They're not actually done anything at all for
a while.
(Bitbake rev: 2300beb50333bb620013b058a7309e7f2042101d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When I originally added this check I didn't quite understand how the
values in this field should be expressed - it seems from reading the
documentation if there is an entry starting with '!' then the list is
a blacklist and we shouldn't expect "linux" to be in the list, or we'll
end up skipping important dependencies.
This fixes fetching the "statsd" npm package.
Fixes [YOCTO #10760].
(Bitbake rev: 7aa6d1586417e0e7d9925917a82caee5884957db)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
An npm package.json file has two dependency fields: dependencies and
optionalDependencies. An item in optionalDependencies *may* also be
listed in dependencies, but this is not required (and not necessary
since if it's in optionalDependencies it will be optional, adding it to
dependencies won't do anything). The code here was assuming that an
optional dependency would always be in both, that's probably because
that was true of the examples I was looking at at the time. To fix it,
just add the optional ones to the list we're iterating over.
(Bitbake rev: c0c50d43266150a80be31ae2c6fcaf37f5ba231d)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a utility function that gives the list of dependent tasks between
two specified tasks (just within the same recipe). This is intended to
be able to be executed from recipe context so it uses the datastore
rather than having access to the runqueue.
This will be used in OpenEmbedded-Core's populate_sdk_ext.bbclass to
get the list of tasks between do_image_complete and do_build.
(Bitbake rev: 433379bf12cf31fdf46defdf66695cf8be9994b1)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
exc_func() fails with a hard to debug exception when the
function does not exist, for example due to a typo:
ERROR: ...: Traceback (most recent call last):
File "/work/bitbake/lib/bb/build.py", line 644, in exec_task
return _exec_task(fn, task, d, quieterr)
File "/work/bitbake/lib/bb/build.py", line 584, in _exec_task
exec_func(func, localdata)
File "/work/bitbake/lib/bb/build.py", line 198, in exec_func
cleandirs = flags.get('cleandirs')
AttributeError: 'NoneType' object has no attribute 'get'
There is code further down which will print a warning, but we don't
reach that unless we allow None as result of of d.getVarFlags() first.
The warning is further down intentionally and has to stay there, to
ensure that 'cleandirs' gets executed also for empty functions.
(Bitbake rev: 25df3db5eeda9dbf417e67e96845f376e5c8e6b3)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was a stra quote character after the word database
and I removed it.
(Bitbake rev: 50afcab054b2d93be96bfd3c224affebcab023cc)
Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes [YOCTO #10886]
Added text descriptions for many of the events in the list of the
"Events" section.
(Bitbake rev: e3b7e8430cb207756b59b32128aa3cef6a626fa1)
Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The cooker is added to the BBServer and then detached creating
a copy of the cooker in the process, if the server isn't in
foreground it cause fd leaks on inotify this can be see using
many instances of tinfoil.
Example:
from bb.tinfoil import Tinfoil
while True:
with Tinfoil() as tinfoil:
input("Pre\n")
tinfoil.prepare(config_only=True)
input("Post\n")
[YOCTO #10873]
(Bitbake rev: 2ada2efb057a291eb34c66d975eb339232137956)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Call explicity close in _writer to avoid fd leakage because isn't
called on Queue.close()
[YOCTO #10873]
(Bitbake rev: 12dfac442d2d5674198485fbeb708a01d215c576)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add ability to limit output per package:
$ bitbake-layers show-appends m4
=== Matched appended recipes ===
m4_1.4.17.bb:
/home/mattsm/git/openembedded-core/meta-selftest/recipes-test/m4/m4_1.4.17.bbappend
Useful for writing tools to process the output, and debugging
bbappends for specific packages
(Bitbake rev: 43668657a8a679acc957e26e6fd8f47ab4cb3da8)
Signed-off-by: Matthew McClintock <msm-oss@mcclintock.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If was pointed out that if we have:
XXX = " A"
XXX_remove_inactive-override = "YY"
then XXX can become "A" and the leading space can be removed. This is because
the remove override code changes the variable value even when there is no
removals active. In the process it dirties the cache.
We don't really need to do this so tweak the code accordingly.
(Bitbake rev: 2bc4d35fb32defc59cd6ed1fc87e35924c201a5c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The idea is that tasks which complete building a recipe (like
do_package_qa) are more important than tasks which start building new
recipes (do_fetch) or those which increase disk usage
(do_compile). Therefore tasks get ordered like this (most important
first, do_rm_work before do_build because the enhanced rm_work.bbclass
was used):
1. ID /work/poky/meta/recipes-support/popt/popt_1.16.bb:do_build
2. ID /work/poky/meta/recipes-core/readline/readline_6.3.bb:do_build
3. ID /work/poky/meta/recipes-connectivity/libnss-mdns/libnss-mdns_0.10.bb:do_build
...
464. ID /work/poky/meta/recipes-sato/images/core-image-sato.bb:do_build
465. ID /work/poky/meta/recipes-graphics/xorg-proto/inputproto_2.3.2.bb:do_rm_work
466. ID /work/poky/meta/recipes-devtools/python/python3_3.5.2.bb:do_rm_work
467. ID /work/poky/meta/recipes-core/packagegroups/packagegroup-base.bb:do_rm_work
...
3620. ID virtual:native:/work/poky/meta/recipes-extended/pbzip2/pbzip2_1.1.13.bb:do_install
3621. ID /work/poky/meta/recipes-devtools/qemu/qemu-helper-native_1.0.bb:do_install
3622. ID /work/poky/meta/recipes-core/zlib/zlib_1.2.8.bb:do_compile_ptest_base
3623. ID /work/poky/meta/recipes-extended/bzip2/bzip2_1.0.6.bb:do_compile_ptest_base
...
3645. ID /work/poky/meta/recipes-support/libevent/libevent_2.0.22.bb:do_compile_ptest_base
3646. ID /work/poky/meta/recipes-core/busybox/busybox_1.24.1.bb:do_compile_ptest_base
3647. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_uboot_mkimage
3648. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_sizecheck
3649. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_strip
3650. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_compile_kernelmodules
3651. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_shared_workdir
3652. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_kernel_link_images
3653. ID /work/poky/meta/recipes-devtools/quilt/quilt-native_0.64.bb:do_compile
3654. ID /work/poky/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb:do_compile
...
The order of the same task between different recipes is the same as
with the speed scheduler, i.e. more important recipes come first.
(Bitbake rev: 70e297e5c285ce0a02e9efd3117ff62cdc77ec12)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The API is required by the revised rm_work.bbclass implementation,
which needs to know all tasks that do_build depends so that it
can properly inject itself between do_build and those tasks.
The new API primarily hides the internal implementation of the "after"
and "before" dependency tracking. Because tasks defined as
precondition via "recrdeptask" may or may not be relevant (they are for
rm_work.bclass), the API also includes support for that.
There's no default value for including recrdeptasks, so developers
have to think about what they need.
(Bitbake rev: 9289ab40e77906e983a2f79cd7602ee95be5025a)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If PATH contains WORKDIR which contains PV which contains SRCPV we can end
up in circular recursion within the fetcher. This code change allows for the recursion
to be broken by giving PV a temporary dummy value in a data store copy.
(Bitbake rev: ce1e70b8018340b54dba3a81d7d379182cb77514)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we're running with bitbake -b, BB_TASKDEPDATA is incorrect and limited.
We really need a way to know this from the metadata and this new variable
provides this in worker context. This means existing code can stop having
to guess.
(Bitbake rev: 05763bc886024dcce2ce6b3060fb00abf79a9402)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are various pieces of code which need to run after the tasks are
finalised but before bitbake locks in on the task dependencies. This
adds such an event so dependency changes in anonymous python can
be accounted for and acted upon by these specific event handlers.
(Bitbake rev: 4dcd0e53f5ff4bf4f2d6cbdc51ff33a5f5f206af)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this, bitbake -b of image recipes cause tracebacks since
the list of providers is empty.
(Bitbake rev: c53e43f3c6a675c0934a7a4e358fd66b049ffca3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently "bitbake universe -c fetch" will error if every task in the system
doesn't have a fetch task. This seems unreasonable so filter the list based
upon whether the task exists. This implementation does assume if a task exits
for a given provider name, it can run, even if the preferred provider that
ends up getting built might be the namespace without the task. This is unlikely
to be a real world issue at this point.
(Bitbake rev: 438986f4e39974930ee3205ca4c4d620090e57bb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton
David Reyna
Belal, Awais
Richard Purdie
Scott Rifenbark
Peter Kjellerstedt
Ross Burton
Elizabeth 'pidge' Flanagan
Diana Thayer
Patrick Ohly
Christopher Larson
Juro Bystricky
Joshua Lock
Andre McCurdy
Matthew McClintock
Alexander Kanavin
Mike Crowe
Anders Darander
Markus Lehtonen
Aníbal Limón
Khem Raj
Robert P. J. Day
Sujith H
brian avery