Commit Graph

5210 Commits

Author SHA1 Message Date
Andre McCurdy c225f0fdb5 bitbake: fetch2: pass --no-same-owner when extracting tar files compressed with 7-Zip
Treat tar files compressed with 7-Zip in the same way as tar files
compressed with other compression formats.

(Bitbake rev: 363a0f54dc7d9930537f0df25173fa31ca1f98ac)

Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-04 23:18:46 +00:00
Andre McCurdy 0788007ccf bitbake: fetch2: remove last remaining usage of deprecated bb.data APIs
(Bitbake rev: 62dc3c3b40a95f9df31f5c917574c3822fb8bb64)

Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-04 23:18:46 +00:00
Andre McCurdy 8f1ed17587 bitbake: fetch2: add initial Amazon AWS S3 fetcher
Class for fetching files from Amazon S3 using the AWS Command Line
Interface. The aws tool must be correctly installed and configured
prior to use.

The class supports both download() and checkstatus(), which therefore
allows S3 mirrors to be used for SSTATE_MIRRORS.

(Bitbake rev: 6fe07ed25457dd7952b60f4b2153d56b15d5eea6)

Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-03 00:19:37 +00:00
Richard Purdie da6652b526 bitbake: fetch2: Allow whitespace only mirror entries
Forcing the use of "\n" in mirror variables is pointless, we can just require that
there are pairs of values.

(Bitbake rev: 044fb04dbe69313ee6908bf4d3cee7f797d0c41c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-03 00:19:37 +00:00
Scott Rifenbark 7abb5b7206 bitbake: bitbake-user-manual: Added "path_spec" parameter to SVN Fetcher
Fixes [YOCTO #10405]

The section on the SVN Fetcher was missing information on the
"path_spec" option.  I added this option and also updated the
examples at the bottom of the section to include that parameter.
Also, made the other two examples consistent.
I also removed the "date" parameter.

Also, updated the "protocol" parameter as well as the "modify"
parameter.  For "modify" I removed the reference to "rsh". I
applied a small wording change to the "protocol" parameter.

Finally, I added a new "ssh" parameter.

(Bitbake rev: 3ce6169afa646ef2b847e5fbabfe0191c93928b7)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-01 15:46:59 +00:00
Scott Rifenbark bcfb0da4fe bitbake: bitbake-user-manual: Rewrite dependency dot file generation
The package-depends.dot and pn-depends.dot files are inaccurate,
missing out key dependencies such those made via the [depends]
flags. As such they can be misleading to the user.

They mainly exist for historical reasons, coming from a time
before we had task based execution.

This commit removes the two dated file formats and replaces
them with a recipe-depends.dot which is a flattened version
of task-depends.dot.

The old format files are removed if present so that the user
can't get confused about why data might not match between files.

The code is also rewritten to use 'with f: f.write()' syntax as
is more commonly used now. Also update the docs to match the change.

(Bitbake rev: f82537d27f2a5bf9d576aa841593db9ec0985ea8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-01 15:46:59 +00:00
Andre McCurdy b16192c938 bitbake: fetch2: don't use deprecated bb.data APIs
Cleanup some more usage of bb.data APIs in the fetchers.

(Bitbake rev: 9752fd1c10b8fcc819822fa6eabc2c1050fcc03b)

Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-01 11:16:07 +00:00
Richard Purdie b1c48fb646 bitbake: bitbake: Update version to 1.33.2
(Bitbake rev: f5ab65610d6d6c0587948b644292f57c07dece0c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-01 11:16:07 +00:00
Peter Kjellerstedt 3004f57a25 bitbake: lib/bb/utils: Add filter()
The bb.utils.filter() function can be used to filter a variable
containing whitespace separated words based on another set of words.
It has been modeled after the bb.utils.contains_any() function.

A typical example of how it can be used is to simplify constructs for
PACKAGECONFIG that depend on DISTRO_FEATURES:

-PACKAGECONFIG ?= "\
-    ${@bb.utils.contains('DISTRO_FEATURES', 'acl', 'acl', '', d)} \
-    ${@bb.utils.contains('DISTRO_FEATURES', 'selinux', 'selinux', '', d)} \
-"
+PACKAGECONFIG ?= "${@bb.utils.filter('DISTRO_FEATURES', 'acl selinux', d)}"

(Bitbake rev: 03ae490366d2046f5b5c185fe4ec2adf1b0a902e)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-01 11:16:07 +00:00
Alexander Kanavin 8dd99c15af bitbake: wget.py: match .lz tarballs as well when determining latest version
(Bitbake rev: c55e09fbd2b64a2b909d9992350b6b2e26e5e86d)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-01 11:16:07 +00:00
Mike Crowe 7b75bf8505 bitbake: process: stop bb.process.communicate mixing bytes and str return types
Python3 regards b"" as False so it is not being converted to a string by
d0f904d407f57998419bd9c305ce53e5eaa36b24. This confusingly causes three
different potential types for each member of the returned tuple.

Let's just assume that everything that's not None is a bytes object and
convert it to a string.

(Bitbake rev: 0cf5589b7fb3582a6caca5014c4d8152347df545)

Signed-off-by: Mike Crowe <mac@mcrowe.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-01 11:16:07 +00:00
Anders Darander 3242826304 bitbake: lib/bb/fetch2: rewrite / to - in package names
Most angular2 packages have names of the form @angular/xxx.
The / obviously can't be used in a file name, replace it with -.

(Bitbake rev: d3bd41d0ec9621307c362b394872b18b8b7ed8d6)

Signed-off-by: Anders Darander <anders@chargestorm.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-01 11:16:07 +00:00
Anders Darander 4b70f27173 bitbake: fetch2/__init__: handle @ in package names
A number of npm packages use @ as a leading chararacter.
Examples are most of the angular2 packages.

(Bitbake rev: 628c4bf6c89b3d62c9b864380b5c8e131a899bff)

Signed-off-by: Anders Darander <anders@chargestorm.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-01 11:16:07 +00:00
Markus Lehtonen 3c83b56309 bitbake: fetch2/svn: change 'rsh' parameter to 'ssh'
The functionality around the 'rsh' parameter seemed to be broken. The
'rsh' parameter was only used when 'protocol' was set to 'svn+ssh' which
is confusing. The 'rsh' parameter was used for setting the value of
'svn_RSH' environment variable, which however, is not supported by svn
(not at least according to SVN documentation).

This patch removes the 'rsh' parameter and replaces it with 'ssh'. This
new (optional) parameter is used when svn+ssh protocol is used and it
can be used to specify the ssh program used by svn. This is achieved by
setting the SVN_SSH environment variable which is mentioned in SVN
documentation.

(Bitbake rev: 5b364b02270b0d7c2b7ca8d67fa2731bf93720ee)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-23 12:50:17 -08:00
Richard Purdie ff734f0560 bitbake: runqueue: Fix collapsed setscene dependency tree
When we removed the postinst sstate dependency handling code from
setscene_depvalid, we noticed things being installed into the old
style sysroot for rootfs tasks which should not have been there,
causing a performance regression.

Analysis revealed that setscene dependencies were "bubbling" over
sstate tasks when they should have been stopping there. The
'continue' added by this patch avoids this issue and eusures
sstate tasks remain contained to their specific chains.

There was another bug in the code this exposed where the acconting
for tasks as they were removed from sq_revdeps was not correct. In
fixing this, what looks like a workaround in another test can then
be simplified.

After this change, populate_sysroot tasks are no longer depending
on package_write_rpm tasks for example, which would make no sense.

A before/after analysis of image dependencies only revealed improved
dependencies after this change.

Recipe specific sysroots did highlight the issue here since the
behaviour of the sysroot dependencies (and processing with depvalid)
was not matching what bitbake itself was doing, with bitbake being
incorrect. Failures were 'safe' in that too many dependencies would
get installed.

(Bitbake rev: 5ef2cb50041fa7106c8de170af73d2a54cb0b1f0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-19 06:28:47 -08:00
Aníbal Limón 10968b4266 bitbake: bitbake-layers: Handle exception raised bytinfoil.prepare()
The tinfoil.prepare method can raise Exceptions when is parsing initial
 data so add this call inside try, finally to avoid get blocked for
don't call tinfoil.shutdown().

The tinfoil_init function was remove because isn't make sense now since
 tinfoil.prepare() needs to be inside try, finally closures.

Example of raised exception and gets blocked:

$ bitbake-layers add-layer ~/repos/meta-openembedded/meta-python/

Traceback (most recent call last):
  File "/home/alimon/repos/poky/bitbake/bin/bitbake-layers", line 83, in
main
    tinfoil.prepare(True)
...
  File "/home/alimon/repos/poky/bitbake/lib/bb/tinfoil.py", line 268, in
run_command
    raise TinfoilCommandFailed(result[1])
bb.tinfoil.TinfoilCommandFailed: Traceback (most recent call last):
  File "/home/alimon/repos/poky/bitbake/lib/bb/command.py", line 81, in
runCommand
    result = command_method(self, commandline)
...
  File "/home/alimon/repos/poky/bitbake/lib/bb/cooker.py", line 1314, in
handleCollections
    raise CollectionError("Errors during parsing layer configuration")
bb.cooker.CollectionError: Errors during parsing layer configuration

(Bitbake rev: 61ebe9a026652e32f9482f66fc3fe7a8672fc093)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-19 06:28:47 -08:00
Paul Eggleton 157d9d92a0 bitbake: tinfoil: ensure PATH includes bitbake's bin directory
The runqueue expects to be able to run 'bitbake-worker' from PATH, but
for example in the OE extensible SDK, tinfoil is used within devtool
where this isn't the case (we're not exposing bitbake to the user, thus
bitbake's bin directory isn't in PATH).

This fixes devtool modify usage within the extensible SDK which has
apparently been broken since the tinfoil2 changes went in.

Fixes [YOCTO #11034].

(Bitbake rev: 52d87e4a58a52174e8e0b6297abaac8fd93ffcc3)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-19 06:28:47 -08:00
Khem Raj b14e61d1bd bitbake: fetch2: Add NFS Stale file exception handling
- In some cases the file descriptor
  is held by nfs client and none of os.path.* is catching
  that, it could mean that error is not doled out because
  client has cached the stat info. In this case we are
  out of luck. Needed to catch IOError, which would be
  causing the Stale error.

- In download method, update_stamp is invoked
  md5sum validation which is found to be throwing
  Stale errors.
- Added error handling to fix the stale errors.

(Bitbake rev: 5a53e7d7b017769a6eb0f0a6335735a1fe51a5ec)

Signed-off-by: Balaji Punnuru <balaji_punnuru@cable.comcast.com>
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-19 06:28:47 -08:00
Robert P. J. Day 6c611d697f bitbake: fetch2: Rename "setup_revisons" to "setup_revisions"
For spelling's sake, rename Python routine "setup_revisons" to
"setup_revisions."

(Bitbake rev: 4df59b027c02ef39d72476251ccd3fd62fc20bf6)

Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-19 06:28:47 -08:00
Robert P. J. Day 1bd4f040a3 bitbake: fetch2: Correct two examples of typo "revsion."
(Bitbake rev: 05f5421b2e44cd58c5912848de43d5884d070150)

Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-19 06:28:46 -08:00
Andre McCurdy 7ab8b6558c bitbake: cooker: detect malformed BBMASK expressions which begin with a separator
When constructing an older style single regex, it's possible for BBMASK
to end up beginning with '|', which matches and masks _everything_.

(Bitbake rev: 56ad67017e601c7e0f6085ca84e29c28d8d4519f)

Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-19 06:28:46 -08:00
Richard Purdie 6cfc1c83b9 bitbake: cooker: Rewrite dependency dot file generation
The package-depends.dot and pn-depends.dot files are inaccurate, missing
out key dependencies such those made via the [depends] flags. As such
they can be misleading to the user.

They mainly exist for historical reasons, coming from a time before
we had task based execution.

This commit removes the two dated file formats and replaces them with
a recipe-depends.dot which is a flattened version of task-depends.dot.

The old format files are removed if present so that the user can't get
confused about why data might not match between files.

The code is also rewritten to use 'with f: f.write()' syntax as is more
commonly used now.

Also update the docs to match the change.

(Bitbake rev: d3e182bc18ff2894f1efc8aad3d508dd432c996e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-19 06:28:46 -08:00
Richard Purdie 701ae7d1f4 bitbake: doc: Replace depexp UI with taskexp and cleanup UI lists
The depexp UI has been replaced with taskexp, also remove old
references to goggle and hob.

(Bitbake rev: 814c2b232ea5f88b5ad5031fb9d389c9c594aa01)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-19 06:28:46 -08:00
Richard Purdie 6a531c8b33 bitbake: ui: Replace depexp with taskexp
The data displayed by the depexp UI is buggy, bitbake can't convey
accurate runtime dependency information.

Convert the UI to become taskexp ("Task Explorer") which provides
a graphical way of exploring the task-depends.dot file. This data
is accurate and being able to visualise the data is an often requested
feature as standard dot file viewers can't usualy cope.

(Bitbake rev: 416a496edc692982af2feec768ac4df8fb799853)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-19 06:28:46 -08:00
Juro Bystricky e98757e267 bitbake: cooker.py: run sanity checks for multiconfig
Also run sanity check for all additional multiconfig configurations
as listed in BBMULTICONFIG, not just the "default" configuration.

[YOCTO#10810]

(Bitbake rev: b9777374cbe63cdcf80767033f628c26d23e90f9)

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-15 20:08:15 -08:00
Sujith H d4b5a70831 bitbake: bitbake: toaster: add wic.bmap support to toaster
Add wic.bmap file extension support for toaster

(Bitbake rev: 6a3f6f1b5cc9ec69b28bebf70daed7579c60cd9e)

Signed-off-by: Sujith H <sujith.h@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-15 20:08:15 -08:00
brian avery 9bc41e1e8d bitbake: toaster: remove cut and paste cruft
There was a superfluous section in toaster start/stop script that was
setting the TOASTER_DIR and had a set of comments around that setting.
This was done in two places and only the last one was effective. This
patch removes the spurious section to make it clearer what was
happening and what TOASTER_DIR was actually getting set to.

(Bitbake rev: e551e73a094f5a1937bce78de9e41908bf3e07c3)

Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-15 20:08:15 -08:00
brian avery f43d97af94 bitbake: toaster: move sqlite database to TOASTER_DIR
The toaster.sqlite database was located in TOASTER_DIR/build.  This
meant that if you named your build directory something else (like cow),
Toaster would fail to make/find the database.  TOASTER_DIR is on the
whitelist unlike BUILDDIR and we need to be able to write there anyway
given our current layout so this should not disrupt anything.

[YOCTO #9992]

(Bitbake rev: 2c48168a86309c0cf2be793e7409a78ba21fca14)

Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-15 20:08:15 -08:00
brian avery 3b7ab16bf8 bitbake: toaster: remove cut and paste cruft
There was a superfluous section in toaster start/stop script that was
setting the TOASTER_DIR and had a set of comments around that setting.
This was done in two places and only the last one was effective. This
patch removes the spurious section to make it clearer what was
happening and what TOASTER_DIR was actually getting set to.

(Bitbake rev: 1f0eac0a172a4fbe1799675f2c3ce989743bd862)

Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-15 20:08:15 -08:00
Richard Purdie f0d5eb39c3 bitbake: lib: Drop now unneeded update_data calls
Now that the datastore works dynamically we don't need the update_data calls
so we can just remove them. They're not actually done anything at all for
a while.

(Bitbake rev: 2300beb50333bb620013b058a7309e7f2042101d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-15 20:08:15 -08:00
Richard Purdie 9f12295626 bitbake: bitbake: Update version to 1.33.1
(Bitbake rev: 95deecabfffdb6a7009ecc385c8b12290fbfcb73)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-07 14:36:01 +00:00
Paul Eggleton 8e61d6d8b7 bitbake: fetch2/npm: fix handling of os field
When I originally added this check I didn't quite understand how the
values in this field should be expressed - it seems from reading the
documentation if there is an entry starting with '!' then the list is
a blacklist and we shouldn't expect "linux" to be in the list, or we'll
end up skipping important dependencies.

This fixes fetching the "statsd" npm package.

Fixes [YOCTO #10760].

(Bitbake rev: 7aa6d1586417e0e7d9925917a82caee5884957db)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-07 14:36:01 +00:00
Paul Eggleton eed25ff021 bitbake: fetch2/npm: handle items only in optionalDependencies
An npm package.json file has two dependency fields: dependencies and
optionalDependencies. An item in optionalDependencies *may* also be
listed in dependencies, but this is not required (and not necessary
since if it's in optionalDependencies it will be optional, adding it to
dependencies won't do anything). The code here was assuming that an
optional dependency would always be in both, that's probably because
that was true of the examples I was looking at at the time. To fix it,
just add the optional ones to the list we're iterating over.

(Bitbake rev: c0c50d43266150a80be31ae2c6fcaf37f5ba231d)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-07 14:36:01 +00:00
Paul Eggleton d01b4bd15b bitbake: lib/bb/build: add tasksbetween() function
Add a utility function that gives the list of dependent tasks between
two specified tasks (just within the same recipe). This is intended to
be able to be executed from recipe context so it uses the datastore
rather than having access to the runqueue.

This will be used in OpenEmbedded-Core's populate_sdk_ext.bbclass to
get the list of tasks between do_image_complete and do_build.

(Bitbake rev: 433379bf12cf31fdf46defdf66695cf8be9994b1)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-07 14:36:01 +00:00
Richard Purdie 6f14790163 bitbake: bitbake: Bump version to 1.33.0 (develoment version)
(Bitbake rev: 2a309cd20cddf3d2258bd00ad06b26452e6cc043)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-27 10:41:09 +00:00
Patrick Ohly 478644aedf bitbake: build.py: avoid exception when function is not defined
exc_func() fails with a hard to debug exception when the
function does not exist, for example due to a typo:

  ERROR: ...: Traceback (most recent call last):
    File "/work/bitbake/lib/bb/build.py", line 644, in exec_task
      return _exec_task(fn, task, d, quieterr)
    File "/work/bitbake/lib/bb/build.py", line 584, in _exec_task
      exec_func(func, localdata)
    File "/work/bitbake/lib/bb/build.py", line 198, in exec_func
      cleandirs = flags.get('cleandirs')
  AttributeError: 'NoneType' object has no attribute 'get'

There is code further down which will print a warning, but we don't
reach that unless we allow None as result of of d.getVarFlags() first.

The warning is further down intentionally and has to stay there, to
ensure that 'cleandirs' gets executed also for empty functions.

(Bitbake rev: 25df3db5eeda9dbf417e67e96845f376e5c8e6b3)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-26 10:44:28 +00:00
Scott Rifenbark a0cc88dc66 bitbake: bitbake-user-manual: Fixed small typo in "Events" section
There was a stra quote character after the word database
and I removed it.

(Bitbake rev: 50afcab054b2d93be96bfd3c224affebcab023cc)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 23:30:15 +00:00
Scott Rifenbark 000d2708b2 bitbake: bitbake-user-manual: Updated Event descriptions
Fixes [YOCTO #10886]

Added text descriptions for many of the events in the list of the
"Events" section.

(Bitbake rev: e3b7e8430cb207756b59b32128aa3cef6a626fa1)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 23:30:14 +00:00
Aníbal Limón 8a3221e6e5 bitbake: lib/bb/main.py: Shutdown cooker when server isn't foreground
The cooker is added to the BBServer and then detached creating
a copy of the cooker in the process, if the server isn't in
foreground it cause fd leaks on inotify this can be see using
many instances of tinfoil.

Example:

from bb.tinfoil import Tinfoil
while True:
    with Tinfoil() as tinfoil:
        input("Pre\n")
        tinfoil.prepare(config_only=True)
        input("Post\n")

[YOCTO #10873]

(Bitbake rev: 2ada2efb057a291eb34c66d975eb339232137956)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 22:38:49 +00:00
Aníbal Limón 509d26f95b bitbake: bb/server/process.py: ProcessEventQueue add close of _writer pipe
Call explicity close in _writer to avoid fd leakage because isn't
called on Queue.close()

[YOCTO #10873]

(Bitbake rev: 12dfac442d2d5674198485fbeb708a01d215c576)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 22:38:49 +00:00
Matthew McClintock a5a51304d1 bitbake: bitbake-layers: show-appends: add optional argument to limit package list
Add ability to limit output per package:

$ bitbake-layers show-appends m4
=== Matched appended recipes ===
m4_1.4.17.bb:
  /home/mattsm/git/openembedded-core/meta-selftest/recipes-test/m4/m4_1.4.17.bbappend

Useful for writing tools to process the output, and debugging
bbappends for specific packages

(Bitbake rev: 43668657a8a679acc957e26e6fd8f47ab4cb3da8)

Signed-off-by: Matthew McClintock <msm-oss@mcclintock.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 22:38:49 +00:00
Richard Purdie 45df694a9f bitbake: data_smart: Fix unneeded variable manipulation
If was pointed out that if we have:

XXX = " A"
XXX_remove_inactive-override = "YY"

then XXX can become "A" and the leading space can be removed. This is because
the remove override code changes the variable value even when there is no
removals active. In the process it dirties the cache.

We don't really need to do this so tweak the code accordingly.

(Bitbake rev: 2bc4d35fb32defc59cd6ed1fc87e35924c201a5c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-22 09:05:39 +00:00
Patrick Ohly bf0ea57e4a bitbake: runqueue.py: revised completion scheduler
The idea is that tasks which complete building a recipe (like
do_package_qa) are more important than tasks which start building new
recipes (do_fetch) or those which increase disk usage
(do_compile). Therefore tasks get ordered like this (most important
first, do_rm_work before do_build because the enhanced rm_work.bbclass
was used):

1. ID /work/poky/meta/recipes-support/popt/popt_1.16.bb:do_build
2. ID /work/poky/meta/recipes-core/readline/readline_6.3.bb:do_build
3. ID /work/poky/meta/recipes-connectivity/libnss-mdns/libnss-mdns_0.10.bb:do_build
...
464. ID /work/poky/meta/recipes-sato/images/core-image-sato.bb:do_build
465. ID /work/poky/meta/recipes-graphics/xorg-proto/inputproto_2.3.2.bb:do_rm_work
466. ID /work/poky/meta/recipes-devtools/python/python3_3.5.2.bb:do_rm_work
467. ID /work/poky/meta/recipes-core/packagegroups/packagegroup-base.bb:do_rm_work
...
3620. ID virtual:native:/work/poky/meta/recipes-extended/pbzip2/pbzip2_1.1.13.bb:do_install
3621. ID /work/poky/meta/recipes-devtools/qemu/qemu-helper-native_1.0.bb:do_install
3622. ID /work/poky/meta/recipes-core/zlib/zlib_1.2.8.bb:do_compile_ptest_base
3623. ID /work/poky/meta/recipes-extended/bzip2/bzip2_1.0.6.bb:do_compile_ptest_base
...
3645. ID /work/poky/meta/recipes-support/libevent/libevent_2.0.22.bb:do_compile_ptest_base
3646. ID /work/poky/meta/recipes-core/busybox/busybox_1.24.1.bb:do_compile_ptest_base
3647. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_uboot_mkimage
3648. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_sizecheck
3649. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_strip
3650. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_compile_kernelmodules
3651. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_shared_workdir
3652. ID /work/poky/meta/recipes-kernel/linux/linux-yocto_4.8.bb:do_kernel_link_images
3653. ID /work/poky/meta/recipes-devtools/quilt/quilt-native_0.64.bb:do_compile
3654. ID /work/poky/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb:do_compile
...

The order of the same task between different recipes is the same as
with the speed scheduler, i.e. more important recipes come first.

(Bitbake rev: 70e297e5c285ce0a02e9efd3117ff62cdc77ec12)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-20 11:52:52 +00:00
Patrick Ohly 633697cce6 bitbake: build.py: add preceedtask() API
The API is required by the revised rm_work.bbclass implementation,
which needs to know all tasks that do_build depends so that it
can properly inject itself between do_build and those tasks.

The new API primarily hides the internal implementation of the "after"
and "before" dependency tracking. Because tasks defined as
precondition via "recrdeptask" may or may not be relevant (they are for
rm_work.bclass), the API also includes support for that.

There's no default value for including recrdeptasks, so developers
have to think about what they need.

(Bitbake rev: 9289ab40e77906e983a2f79cd7602ee95be5025a)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-20 11:52:52 +00:00
Richard Purdie 3f8d56d113 bitbake: fetch2: Avoid recursive errors
If PATH contains WORKDIR which contains PV which contains SRCPV we can end
up in circular recursion within the fetcher. This code change allows for the recursion
to be broken by giving PV a temporary dummy value in a data store copy.

(Bitbake rev: ce1e70b8018340b54dba3a81d7d379182cb77514)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-20 11:52:52 +00:00
Richard Purdie 51abd41ff6 bitbake: cooker: Add BB_LIMITEDDEPS support
When we're running with bitbake -b, BB_TASKDEPDATA is incorrect and limited.
We really need a way to know this from the metadata and this new variable
provides this in worker context. This means existing code can stop having
to guess.

(Bitbake rev: 05763bc886024dcce2ce6b3060fb00abf79a9402)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-20 11:52:52 +00:00
Richard Purdie 631229df72 bitbake: event/ast: Add RecipeTaskPreProcess event before task finalisation
There are various pieces of code which need to run after the tasks are
finalised but before bitbake locks in on the task dependencies. This
adds such an event so dependency changes in anonymous python can
be accounted for and acted upon by these specific event handlers.

(Bitbake rev: 4dcd0e53f5ff4bf4f2d6cbdc51ff33a5f5f206af)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-20 11:52:52 +00:00
Richard Purdie d8b2257f57 bitbake: runqueue: Fix traceback when using -b
Without this, bitbake -b of image recipes cause tracebacks since
the list of providers is empty.

(Bitbake rev: c53e43f3c6a675c0934a7a4e358fd66b049ffca3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-20 11:52:52 +00:00
Richard Purdie 918736d6c9 bitbake: cooker: Only add universe targets where the task exists
Currently "bitbake universe -c fetch" will error if every task in the system
doesn't have a fetch task. This seems unreasonable so filter the list based
upon whether the task exists. This implementation does assume if a task exits
for a given provider name, it can run, even if the preferred provider that
ends up getting built might be the namespace without the task. This is unlikely
to be a real world issue at this point.

(Bitbake rev: 438986f4e39974930ee3205ca4c4d620090e57bb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-20 11:52:52 +00:00
Juro Bystricky 7d18d05d45 bitbake: cooker.py: new multiconfig '*' syntax support
Currently you cannot build a target for all the configured multiconfigs without
specifying a list. The list can be quite long, requiring to type several lines
of text.

This enhancement is to support globbing so that you can do this,
e.g. instead of:

    $ bitbake multiconfig:A:bash multiconfig:B:bash bash

you can do:

    $ bitbake multiconfig:*:bash

There are real world use cases where it is desirable to use multiconfig with
two different tasks. For example:  SDKs with multiple toolchains but also
containing set of additional host tools, or multiconfig builds requiring one image for
the  main CPU(s) and a different co-image for a companion CPU.
For this reason, two variations of the new syntax are supported.

For example, the following:

    $ bitbake multiconfig:*:meta-toolhchain

would expand to:

    $ bitbake multiconfig:A:meta-toolchain multiconfig:B:meta-toolchain meta-toolchain

However the following:

    $ bitbake multiconfig:*:meta-toolhchain hosttools

would expand to:

    $ bitbake multiconfig:A:meta-toolchain multiconfig:B:meta-toolchain hosttools

In other words, if the user specified the "default" task explicitly, it replaces the implicit
"default" task.

[YOCTO#10680]

(Bitbake rev: 1559ad1fc3ae8b1e4e75b210f598b90be192571b)

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-19 20:20:16 +00:00