Commit Graph

228 Commits

Author SHA1 Message Date
Joshua Lock c4e2c59088 meta: remove True option to getVar calls
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.

Search made with the following regex: getVar ?\(( ?[^,()]*), True\)

(From OE-Core rev: 7c552996597faaee2fbee185b250c0ee30ea3b5f)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:23 +00:00
Richard Purdie f6b0c60664 sstate: Ensure we don't remove sigbasedata files
We don't remove sigdata files, we also shouldn't remove sigbasedata files
as this hinders debugging.

(From OE-Core rev: 1ebd85f8dfe45b92c0137547c05e013e340f9cec)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-04 12:50:55 +00:00
Aníbal Limón f205384495 classes/sstate.bbclass: Enable thread lock when checkstatus
The checkstatus function fires an event to notify bitbake UI about
the progress of the task, this function is implemented using ThreadPool
and is causing event lose when multiple threads tries to fire an event
(writes over socket/fd).

[YOCTO #10330]

(From OE-Core rev: 6e0bb9d141438c0051c32b0d3a247915b71ccb82)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-05 10:10:11 +01:00
Ulf Magnusson 39f80cefcc sstate.bbclass: Use bb.fatal() instead of raising FuncFailed
This sets a good example and avoids unnecessarily contributing to
perceived complexity and cargo culting.

Motivating quote below:

< kergoth> the *original* intent was for the function/task to error via
           whatever appropriate means, bb.fatal, whatever, and
           funcfailed was what you'd catch if you were calling
           exec_func/exec_task. that is, it's what those functions
           raise, not what metadata functions should be raising
< kergoth> it didn't end up being used that way
< kergoth> but there's really never a reason to raise it yourself

FuncFailed.__init__ takes a 'name' argument rather than a 'msg'
argument, which also shows that the original purpose got lost.

(From OE-Core rev: 33611b69c221cf875eba1c7cb599c256825ae470)

Signed-off-by: Ulf Magnusson <ulfalizer@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-04 16:29:04 +01:00
Richard Purdie 0573dceb7b sstate: Add postrm command support
There are some issues in sstate which can't be handled by file removal
alone. Currently there is no way to execute a command against sstate and
doing so is potentially problematic for things like dependencies. This
patch adds a mechanism where any "postrm" script is executed if its present
allowing some openjade/sgml issues to be resolved.

[YOCTO #8273]

(From OE-Core rev: 2268efd0cd3ddb40870c4c424d10444ba86d2849)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-16 15:24:02 +01:00
Max Krummenacher f5e2dafcb8 sstate.bbclass: normalize whitelisted paths
Without this a whitelisted path containing /../ breaks the test for a file
allowed to be provided by more than one recipe.

Noticed when local.conf contains:
DEPLOY_DIR = "${TOPDIR}/../deploy"

|ERROR: core-image-minimal-1.0-r0 do_image_complete: The recipe
| core-image-minimal is trying to install files into a shared area when those
| files already exist. Those files and their manifest location are:
| .../poky/deploy/images/qemux86/README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt
| Matched in b'manifest-qemux86-linux-yocto.deploy'

(From OE-Core rev: d82acf20541463b14a811788d28fb1db3539885b)

Signed-off-by: Max Krummenacher <max.krummenacher@toradex.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-14 22:22:11 +01:00
Richard Purdie bc31120ec6 sstate: Avoid duplicate README file errors for sdk under sstate control
(From OE-Core rev: 4bd3a90c8fb034b4d899d0560d75d81f56e27e0a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-04 00:07:29 +01:00
Ed Bartosh 180eebf976 sstate.bbclass: skip packaging if SSTATE_SKIP_CREATION is set
SSTATE_SKIP_CREATION variable will be used to skip creation of
sstate .tgz files. It makes sense for image creation tasks as
tarring images and keeping them in sstate would consume a lot of
disk space.

(From OE-Core rev: 7e821ccd221916ae8482b9113df2de704f4a99a4)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 23:45:53 +01:00
Paul Eggleton f84b01b289 meta-extsdk-toolchain: add meta-recipe to install toolchain into eSDK
Add a meta-recipe to bring the toolchain into the extensible SDK. This
was modelled on meta-ide-support but some adjustments were needed to the
dependency validation function in sstate.bbclass to ensure that all of
the toolchain gets installed into the sysroot. With this, after
installing a minimal eSDK you only need to run the following after
sourcing the environment setup script to get the toolchain:

  devtool sdk-install meta-extsdk-toolchain

Addresses [YOCTO #9257].

(From OE-Core rev: 8110806b1b5534ae830a4fdd1a5293c86a712d0b)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:56:31 +01:00
Ed Bartosh 6ac1f9038d sstate.bbclass: make PV to depend on PV variable value
Currently PV is defined in meta/conf/bitbake.conf as a python
expression: "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE',
False),d)[1] or '1.0'}". As FILE is whitelisted it causes PV to
not depend on it. This causes sstate code to not detect that
PV changes when recipe filename changes.

Making PV to explicitly depend on PV variable value overrides default
behaviour. Instead of depending on python expression bitbake depends
on evaluated value of PV variable, which should fix the above
mentioned issue.

[YOCTO #9806]

(From OE-Core rev: 918646ca803d56004fb0ab7c21e86cc9cb14513d)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-20 10:28:45 +01:00
Paul Eggleton 6677dd37ad classes/sstate: add a mode to error if sstate package unavailable
If BB_SETSCENE_ENFORCE is set to "1" and an sstate package fails to
download outside of the whitelist specified by
BB_SETSCENE_ENFORCE_WHITELIST, then fail immediately so you can tell
that the problem was caused by failing to restore the task from sstate.

Part of the implementation of [YOCTO #9367].

(From OE-Core rev: 9e711b54487c3141d7264b8cf0d74f9465020190)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:23 +01:00
Paul Eggleton e769dce794 classes/sstate: show progress during sstate object availability check
Show progress through bitbake's standard terminal UI when checking for
shared state object availability, since this can take some time if there
are a large number of tasks to be executed and/or the network connection
is slow.

Part of the implementation for [YOCTO #5853].

(From OE-Core rev: 1a064385d6921ec90b33c9064dafaab11a36267c)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:23 +01:00
Richard Purdie 297438e965 classes/lib: Convert to use python3 octal syntax
The syntax for octal values changed in python3, adapt to it.

(From OE-Core rev: 737a095fcde773a36e0fee1f27b74aaa88062386)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:00 +01:00
Randy Witt 1cdd9795dd sstate.bbclass: Don't create symlinks, download to the correct location
Previously the sstate was all downloaded to the same directory and then
symlinks were added in the directories that pointed to the siginfo and
sstate in the parent directory.

This change makes it so that now the files are just downloaded to the
correct location without the need for symlinks.

(From OE-Core rev: 55d25ed6b30ed7105d3b6421fbf2a03cea009a59)

Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-25 07:50:17 +01:00
Richard Purdie 7db0cc27d5 sstate: Ensure we sort the value used for SSTATETASKS for determinism
This avoids 'basehash changed' errors with python 3 but could break
build determinism in general.

(From OE-Core rev: 488f88fc12eea9788e5db1de354f0118e2c58878)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-17 14:43:28 +01:00
Randy Witt 7e902807cb sstatesig.py: Split single locked sigs check into multiple checks
Add the SIGGEN_LOCKEDSIGS_TASKSIG_CHECK and
SIGGEN_LOCKEDSIGS_SSTATE_EXISTS_CHECK variables to replace
SIGGEN_LOCKEDSIGS_CHECK_LEVEL.

SIGGEN_LOCKEDSIGS_TASKSIG_CHECK will no control whether there is a
warning or error if a task's hash in the locked signature file doesn't match
the computed hash from the current metadata.

SIGGEN_LOCKEDSIGS_SSTATE_EXISTS_CHECK will control whther there is a
warning or error if a task that supports sstate is in the locked
signature file, but no sstate exists for the task.

Previously you could only have warning/errors for both controlled by
SIGGEN_LOCKEDSIGS_CHECK_LEVEL. This was an issue in the extensible sdk,
because we know sstate won't exist for certain items in the reverse
dependencies list for tasks. However, we still want to error if task
signatures don't match.

[YOCTO #9195]

(From OE-Core rev: 0fe2a5e5ffd01e926d0f3d4c78ad9910296e2d1a)

Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-09 23:00:43 +01:00
Maxin B. John cbe33ecb15 useradd.bbclass: remove user/group created by the package in clean* task
At present, if a recipe is built which creates users/groups via
useradd.bbclass, those users/groups are not removed from sysroot
when the recipe/package is cleaned using clean/cleansstate/cleanall
or when a recipe is rebuild and 'unstaged' from the the sysroot.

The "userdel_sysroot_sstate()" provides that functionality.

[YOCTO #9262]

(From OE-Core rev: b5304ce438666a7418746f4ddd32703ae3188089)

Signed-off-by: Maxin B. John <maxin.john@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-06 22:57:25 +01:00
Richard Purdie ea53d1e4a9 sstate: Allow late expansion of NATIVELSBSTRING
uninative needs to adjust NATIVELSBSTRING fairly late in the
configuration parsing process but the sstate code encodes it into
variables. Since this string doesn't vary on a per recipe basis, we
defer its expansion until usage time.

(From OE-Core rev: fb680ab8b3cf7aba9b579403b1aeb96d30762320)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 11:55:37 +00:00
Richard Purdie 78b6109440 cml1/sstate: Fix missing getVar parameter
(From OE-Core rev: 84065e9b33e6d401e4e50436665838be2d0267ad)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-19 15:33:02 +00:00
Markus Lehtonen 4699483773 sstate.bbclass: use oe.gpg_sign for gpg signing
[YOCTO #9006]

(From OE-Core rev: a3916b29c8c3d0b1570c40c0ed3a1a15c43925bb)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-19 01:01:25 +00:00
Ming Liu 14e93850be sstate: Add ca-certificates-native to postinst recipes list
(From OE-Core rev: a6426479f6348524a904ba28c16c1e3c8793cb08)

Signed-off-by: Ming Liu <peter.x.liu@external.atlascopco.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-16 09:04:23 +00:00
Richard Purdie ae2f3a309f sstate: Ensure populate_lic sstate objects are cleaned
do_cleansstate wasn't cleaning do_populate_lic sstate objects in the
native/cross case since the wildcard path entry wasn't being cleared
at the same time as the path extra prefix. Fix by clearing it at the
same time.

(From OE-Core rev: af72c507f5e20acf64b7431cee989af9908ef199)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 16:06:24 +00:00
Richard Purdie 9fd214d2e7 sstate: Fix SSTATE_SWSPEC only used by populate_lic tasks
This variable is only used by populate_lic tasks (gcc-source overrides it)
and refers to BPN. In recipes like gcc, where there are multiple variants,
it is resulting in sstate objects which encode PN (they install into a PN subdir)
but the sstate object reflects BPN. This leads to corruption between then and
eventually, warnings from image builds like:

WARNING: The license listed GPLv3 was not in the licenses collected for recipe gcc
WARNING: The license listed GPL-3.0-with-GCC-exception was not in the licenses collected for recipe gcc
WARNING: The license listed GPLv3 was not in the licenses collected for recipe gcc
WARNING: The license listed NCSA was not in the licenses collected for recipe gcc-sanitizers
WARNING: The license listed MIT was not in the licenses collected for recipe gcc-sanitizers
WARNING: The license listed GPL-3.0-with-GCC-exception was not in the licenses collected for recipe gcc
WARNING: The license listed GPLv3 was not in the licenses collected for recipe gcc
WARNING: The license listed NCSA was not in the licenses collected for recipe gcc-sanitizers

By referring to PN, as used by license.bbclass, this issue is resolved.

(From OE-Core rev: dd4a319efcfa1174ba0f48fd15cb446280c426e4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-04 23:39:00 +00:00
Richard Purdie 5503a22ad0 sstate: Revert using -m option to tar in sstate
In http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=2d89cff42af2bb0049224bfaaebaa2b21966169f
we added a workaround for dealing with lack of time sync between build
machines and their users.

This has turned out to cause problems for people who rely on timestamps
being preserved in sstate output.

Since our autobuilders are all in time sync with ntp, revert the commit.

[YOCTO #8996]

(From OE-Core rev: 004a8bffdb501a99530dab2bc1c5f3d44b97b8f7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-01 11:20:19 +00:00
Richard Purdie 2f1bcc1915 sstate: Add back packagedata on packagedata dependencies
(From OE-Core rev: 654f0eec426e882e50f688f6d097d992e34e5b40)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-30 11:37:04 +00:00
Richard Purdie 5410aff5a9 sstate: Improve handling of useradd dependencies
useradd has sstate [depends] for both do_package and do_populate_sysroot
yet the dependency validation code only covers do_package.

Add coverage of populate_sysroot, else the order inversion that [depends]
creates means unexpected installation of users of useradd.bbclass (e.g.
avahi do_populate_sysroot) in cases where it shouldn't be (e.g.
libnss-mdns -c packagedata).

The code needs to move above the other populate_sysroot intercept code
since there are specific cases we need to cover before that code.

The result of this change is more optimal installation of sstate objects
in common usage scenarios.

(From OE-Core rev: 21368b4a7934b3a199e61c1072416fd5c091e5ff)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-30 11:37:03 +00:00
Ross Burton 258676bfa7 sstate: display the sysroot name when cleaning for clarity
When cleaning old builds from the sysroots, also print the sysroot architecture.

(From OE-Core rev: 98a98a4bf71f74b5c27291ee9a2ac14006377e49)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-24 09:40:21 +00:00
Ross Burton 18357688e9 sstate: replace verbose manifest removal with a single count
If there are lots of changes between the previous build and the build about to
start bitbake will potentially print pages of:

  DEBUG: Stamp $BUILD/stamps/corei7-64-poky-linux/libdrm/2.4.66-r0 is not reachable, removing related manifests

Instead of spamming the console with this list, write the list of manifests only
to the debug log and simply write a count to the console.  This way the user
doesn't get spammed but still knows what is happening if bitbake appears to
stall with heavy I/O.

(From OE-Core rev: e777b541c76dad293d1c214e46c00f8f78fe0539)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-19 17:37:35 +00:00
Richard Purdie 9e867efdc8 sstate: Add packagedata to list of tasks not to recurse
If we "bitbake X -c packagedata" and the packagedata comes from sstate, we
don't need any of the tasks dependencies. This is similar to the
populate_lic case, we only care about the end result.

Therefore short circuit the dependencies so packagedata doesn't pull in
any other dependencies.

(From OE-Core rev: 76aa4f49db7d32bbd35703c64470fdfc63f403a4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-19 17:24:53 +00:00
Richard Purdie 5003d14761 sstate: Ensure populate_lic dependencies are not followed
It was observed that do_fetch dependencies (e.g. subversion-native of tremor)
were being installed even when sstate was available and hence no fetch was
needed. This turned out to be due to the recursive nature of the rootfs
dependencies which include populate_lic.

We can explicitly whitelist these dependencies as being ones we don't
need to follow when installing sstate (the code defaults to being paranoid).

[YOCTO #8746]

(From OE-Core rev: 06c891f0fa6c7712eae233dd18612f1ab1889c45)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 22:30:57 +00:00
Richard Purdie 03f15e5199 sstate: Ensure siginfo and sig files are also touched
Maintaining an sstate mirror is complicated at the moment as only the main
sstate tgz files have their timestamp updated when they're used.

This causes problems as the siginfo files can be removed from the server
due to inactivity but the sstate fetch code tries to fetch them leading
to confusing warnings and reduced debug capability.

This change ensures we touch any sig/siginfo files present and should
help ageing of the objects on sstate mirrors. It doesn't handle the intermediate
task siginfo files but those are never fetched by the current sstate code
so are an order of magnitude less problematic.

(From OE-Core rev: 80b3974081c4a8c604e23982a6db8fb32c616058)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-11-25 08:09:00 +00:00
Ross Burton 59b27d558f sstate: respect GPG_BIN and GPG_HOME
The package feed signing code supports the user providing the path to the gpg
binary and an alternative gpg 'home' (usually ~/.gnupg), which are useful for
both deployment and QA purposes.

Factor out the gpg command line construction to a function which can fetch both
of these variables, and also use pipes.quote() to sanitise the arguments when
used in a shell context.

[ YOCTO #8559 ]

(From OE-Core rev: 6daf138822bbbc46960121d3b76b42eaf19e7c0e)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-11-24 15:50:26 +00:00
Ross Burton 5fd30897f5 sstate: run recipe-provided hooks outside of ${B}
To avoid races between the sstate tasks/hooks using ${B} as the cwd, and other
tasks such as cmake_do_configure which deletes and re-creates ${B}, ensure that
all sstate hooks are run in the right directory, and run the prefunc/postfunc in WORKDIR.

(From OE-Core rev: 1d3bde02641f4b40030cf7e305ee3d7c2faabe29)

(From OE-Core rev: a026dbfa0ce4ee769ae2c714f27f4c4eaa4a0c73)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-28 12:00:22 +01:00
Paul Eggleton 3a9e230b7a classes/sstate: break out function to get sstate manifest filename
It is useful in a few different contexts to see which files have been
written out by an sstate task; break out a function that lets us get the
path to the manifest file easily.

(From OE-Core rev: 090196dd2d8f4306b34b239e78c39d37cc86034c)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-12 22:48:45 +01:00
Ross Burton 48a144b4a4 sstate: only fetch remote signatures if verification is enabled
Only fetch remote signatures if verification has been enabled, as otherwise the
fetcher throws errors that sstate.bbclass can't ignore.

[ YOCTO #8265 ]

(From OE-Core rev: aa8ba5c4af2636dbc2d0c1ad9c230e8aa26d9962)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-12 22:48:35 +01:00
Ross Burton e3feac122b sstate: implement basic signing/validation
To provide some element of integrity to sstate archives, allow sstate archives
to be GPG signed with a specified key (detached signature to a sidecar .sig
file), and verify the signatures when sstate archives are unpacked.

(From OE-Core rev: 237b6c51b42b0c64434dc45685e10f757ac939c2)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-06 15:26:27 +01:00
Jussi Kukkonen 5e7f26d082 gtk-icon-utils: Rename, upgrade, add binary
* Rename from gtk-update-icon-cache to gtk-icon-utils to better
  reflect the content. Fix references in other recipes and classes
* Upgrade to GTK+ 3.16.4
* Add gtk-encode-symbolic-svg binary: it is used by icon themes
  (e.g. Adwaita) to generate png versions of svg icons.
* Depend on librsvg-native for gtk-encode-symbolic-svg
* Add a patch that removes Gdk dependency from gtk-encode-symbolic-svg:
  this way the native build stays slim.

(From OE-Core rev: 5f4fe36f735ff8b97c5112b16bd90e794053573d)

Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-07-16 20:39:15 +01:00
Andre McCurdy 7e7e1239d5 sstate.bbclass: remove redundant cd and mkdir commands
Update sstate_create_package and sstate_unpack_package to remove
redundant initial mkdir and cd commands. The working directory is
now setup correctly before the shell functions are called.

(From OE-Core rev: fd2d50d46e38d86829f88b5884a6fde6c38f155e)

Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-07-16 15:09:25 +01:00
Aníbal Limón 7f8286fb9e classes/sstate.bbclass: Improve checkstatus using connection cache.
Use FetcherConnectionCache to improve times when do checkstatus over
sstate resources.

[YOCTO #7796]

(From OE-Core rev: e6f66370c6ce15aca18ef64491bab3dc92b80c57)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-07-12 22:55:46 +01:00
Aníbal Limón 615b351bf4 oe/utils.py: Add support for init/end helper functions in ThreadWorker.
Add init/end helper functions for ThreadWorker also pass ThreadWorker
as first argument to init/end/func functions this enables per-thread
storage handling.

classes/sstate.bbclass: Add thread_worker argument to checkstatus
function.

(From OE-Core rev: 08c50d62b520c8405f034e3d7adeea89e06226ee)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-06-26 09:27:30 +01:00
Aníbal Limón 3fa32158c4 oe/utils.py: Fix thread leakage in ThreadPool
In order to fix Thread leakage caused by not call join() in Threads,

Pass num_tasks in ThreadPool for add all the tasks into a Queue this
enable catch of Queue.Empty exception and exit the threads.

classes/sstate.bbclass: Change checkstatus function to match new
ThreadPool operation.

(From OE-Core rev: 524d92ed7b53bef933527095e82f378b934f25ef)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-06-26 09:27:30 +01:00
Richard Purdie 86d30d756a meta: Add explict getVar param for (non) expansion
Rather than just use d.getVar(X), use the more explict d.getVar(X, False)
since at some point in the future, having the default of expansion would
be nice. This is the first step towards that.

This patch was mostly made using the command:

sed -e 's:\(getVar([^,()]*\)\s*):\1, False):g' -i `grep -ril getVar *`

(From OE-Core rev: ab7c1d239b122c8e549e8112c88fd46c9e2b061b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-06-23 11:57:25 +01:00
Richard Purdie ccb2d1b481 sstate: Add eventhandler which cleans up stale recipe data
"Incremental builds do not work well when renaming recipes or changing
architecture" is a long standing issue which causes people considerable
pain. We've struggled for a long time to come up with a way to
generically address the problem.

There are additional issues where removal of a layer caused data to
continue to exist and additionally, changing DISTRO_FEATURES also caused
problems in an existing TMPDIR.

This patch attempts to address this by adding a mapping between stamp
files and manifests. After parsing we can easily tell which stamp files
are still reachable, if any manifest has a stamp that can no longer be
reached, we can remove it. Since this code ties this to the sstate
architecture list, it will not remove data from other than the current
MACHINE (and its active architectures). It does not clean the sstate
cache so if another build activates something which was cleaned, it
should reinstall from sstate.

We can also go one step further, depending on the setting of
SSTATE_PRUNE_OBSOLETEWORKDIR, workdirs which are no longer active can
also be removed. This avoids the buildup of many old copies of data in
WORKDIR for example when versions are upgraded.

The one thing which may surprise people with this change is if you
remove a layer, data added by that layer will be "uninstalled" before
the next build continues. I believe this is a feature and a good thing
to do though.

This code is safe with existing builds. If something isn't in the new
index it simply isn't removed. Since changes to the sstate code trigger
a rebuild, after this merges, we can assume the code will start to
detect changes from that point onwards.

[YOCTO #4102]

(From OE-Core rev: 4ea39427eedeadd51439a62fa015c86be30c3445)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-06-10 12:00:13 +01:00
Richard Purdie 801436bf31 sstate: Add note when performing sstate mirror checks
The build can pause whilst remote sstate mirrors are checked for
sstate objects. Inform the user this is happening.

(From OE-Core rev: 4f4cfe8a4a0c602d6e11ea9bb75b52241deda9d6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-06-03 16:38:45 +01:00
Richard Purdie 1f839f233e sstate: Parallelise checkstatus calls for sstate mirror
Currently the urls are checked serially which is a performance bottleneck when
looking at http:// urls in particular. This adds code to check the url status in
parallel, mirroring the way we do this elsewhere.

We need the datastore for the fetcher so we use threads, not multiprocess.

(From OE-Core rev: 77c4865bbde4cd2a061cf333f9ad798afc6de0ef)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-06-03 16:38:45 +01:00
Richard Purdie 5bcf9666a6 sstate: Add handling of do_shared_workdir task
Changing TMPDIR and rebuilding an image was resulting in rebuilds of
kernels due to dependencies on the shared_workdir task. If installed
from sstate, nothing needs this task so add it to the whitelisted
task patterns.

After this change, the kernel does not rebuild when a new TMPDIR and
hot sstate cache is used.

(From OE-Core rev: 624f575298c74272b69560735cb6f13f5a7761c1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-05-21 10:51:15 +01:00
Richard Purdie 1237db6ee3 sstate: Improve HASHCHECK function to make siginfo configurable
In some cases we want to test the availability of siginfo files, in some
cases we do not and really want the .tgz files (which may or may
not be present too). This makes adds a parameter to the function to allow
this.

(From OE-Core rev: e8867b97a772f8e64fe209436c15d382b2175ca0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-05-15 17:44:59 +01:00
Richard Purdie 591e6a6dcb sstate: Use SSTATE_DIR for FILESPATH
FILESPATH was only being overridden in one fetch location, it should be
equally handled in both.

Also use SSTATE_DIR as FILESPATH so that mirror urls which do remapping
can search the local SSTATE_DIR for other paths.

Also ensure that MIRRORS is removed in both locations, previously
it was only unset in one but both codepaths should be consistent.

(From OE-Core rev: ab6bebddbdefec323e284b6438d9c57b3d8a2cc3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-04-29 10:58:54 +01:00
Adrian 6ad040a522 Add archiver folder to SSTATE_DUPWHITELIST
Without this patch the source archiver class is not allowed to
archive the same source archive for different builds.

(From OE-Core rev: 37284185912d496a974d507bde7208d93eefc29d)

Signed-off-by: Adrian <adrian.freihofer@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-03-20 11:03:45 +00:00
Peter Urbanec 754288c387 sstate.bbclass: Fix up white space lost in last commit.
Commit e9672387 split one long line into a multi-line string, but in
the process white space between words was lost. This results in badly
formatted output when this message is printed.

(From OE-Core rev: b145374c0a498de0160a9b81f50ce0066ab14862)

Signed-off-by: Peter Urbanec <openembedded-devel@urbanec.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-10-18 16:14:04 +02:00