Commit Graph

5162 Commits

Author SHA1 Message Date
Paul Eggleton 4794518720 classes/copyleft_filter: include AGPL in default included list
If we're including all GPL licenses then we should include AGPL in
the default COPYLEFT_LICENSE_INCLUDE since it has similar terms (and
currently we do have one AGPL-licensed recipe in OE-Core, namely db).

(Note that the default list isn't meant to be a guaranteed complete
safe list - seek legal advice based on what you're distributing if
unsure.)

(From OE-Core rev: a16fbf6970056a79cac8b70ce54e55e61ef63a8d)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-15 09:29:56 -08:00
Andrea Adami 0fce3ec656 kernel.bbclass: Make symbol to vmlinuz.bin in boot directory
The changes committed with 6e58f54be103814b6b8a85b236510633c49e6832 did only
consider the vmlinuz ELF kernel but some devices do reqire a raw vmlinuz.bin
(as for example gcw0 in meta-handheld booted by ubiboot).
Expand the above mentioned patch to consider this case.

(From OE-Core rev: 413a5faf99280f4e083358e4134aebbcfa47bb4c)

Signed-off-by: Andrea Adami <andrea.adami@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-15 09:29:56 -08:00
Jussi Kukkonen 0d637857f2 cve-check.bbclass: Fix dependencies
With recipe-specific sysroots the cve_check task must depend on
cve-check-tool-native:do_populate_sysroot to get the cve-check-tool
binary into the recipe sysroot.

A normal DEPENDS isn't used to avoid cyclic dependencies.

(From OE-Core rev: bd60b1018bc0304bc928701e6d1090c8b1223616)

Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-15 09:29:56 -08:00
Juro Bystricky b1f99066bd testimage: Implement custom target loading
This patch implements custom target loading for testimage, currently
missing due to major changes to the test framework.
Custom targets can be defined in various meta-layers, so we
need an extra path information in order to find them.
Any other additional info is retrieved as usual via the variables
TEST_TARGET and TEST_SUITES

(From OE-Core rev: 1dc8010afd71fe46fb28bb86fb7c07a5fbd3d7cf)

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-15 09:29:55 -08:00
Richard Purdie bfd0a39bdf classes: Drop now unneeded update_data calls
Now that the datastore works dynamically we don't need the update_data calls
so we can just remove them. They're not actually done anything at all for
a while.

(From OE-Core rev: 8de0c5d3bd01919e2bf0394f9c485936d6098cec)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-15 09:29:55 -08:00
Richard Purdie 3eee8e99e1 sstate: Make absolute symlinks an error
The current relocation code is broken, at least in the native case. Fixing it
would mean trying pass in new data on sstate tasks about the relative positioning
of symlinks compared to the sstate relocation paths. Whilst we could do this,
right now I'm favouring making this an error and fixing the small number of
problematic recipes we have in OE-Core (3).

(From OE-Core rev: cf94de4ddee3e5072da8608c9151301fcec02cd0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-09 10:52:03 +00:00
Richard Purdie 23d9886aae openssl/fontconfig/bzip2: Use relative symlinks instead of absolute ones (using a new class)
Absolute path symlinks are a bit of a pain for sstate and the native versions
of these recipes currently contain broken symlinks as a result. There are
only a small number of problematic recipes, at least in OE-Core, namely the
three here.

Rather than trying to make sstate handle this magically, which turns out to
be a harder problem than you'd first realise, simply make the symlinks relative
early in the process and avoid all the problems.

The alternative is adding new complexity to sstate which we could really
do without as without the complexity, you can't always tell where the
absolute symlink is relative to (due to prefixes used for native sstate).

(From OE-Core rev: e478550c8cd889f12e336e268e9e3b30827bf840)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-09 10:52:03 +00:00
Andre McCurdy 1baae2a0d1 image_types.bbclass: fix image dependency chain collection
If image type "foo" depends on image type "bar.xz", then dependencies
should be collected from the base image type (ie "IMAGE_DEPENDS_bar")
not from "IMAGE_DEPENDS_bar.xz".

(From OE-Core rev: 8a9f249a9166347cc0468191ce130003e3d306e1)

Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-07 14:50:10 +00:00
Andreas Oberritter 36e7717efe kernel-module-split: Allow custom suffix for package names
This makes it possible to restore the behaviour changed by commit 78cde87
"kernel-module-split: Append KERNEL_VERSION string to kernel module name".

(From OE-Core rev: 4aabdddf67859cd8518e5cb672fb391c859817e0)

Signed-off-by: Andreas Oberritter <obi@opendreambox.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-07 14:50:10 +00:00
Paul Eggleton 21ace8b620 classes/npm: set HOME during do_install
In do_compile we set HOME so that ~/.npm* only get created in the work
directory; we need to do the same in do_install as well or they'll go
into the user's home directory which we do not want.

(From OE-Core rev: a667cb9ed2dceb804b85d1a024e2619bc34d2681)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-07 14:50:09 +00:00
Paul Eggleton 014278d370 classes/populate_sdk_ext: account for custom image tasks
Any custom tasks that were added on the image between do_image_complete
and do_build were not being taken into account. Use the newly added
bb.build.tasksbetween() function to take care of that.

(From OE-Core rev: c966dee6208d3b7a9fa3c1b8aa1266058dcb65ff)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-07 14:50:09 +00:00
Paul Eggleton 1a9f33ccc7 classes/patch: switch to new bb.build.tasksbetween() function
A generic version of the code to work out the tasks between two
tasks (based on the code here) has been added to bb.build, so use that
instead.

(From OE-Core rev: 1226e2eb2678407c315befd5700e653602e616a0)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-07 14:50:09 +00:00
Paul Eggleton 4b870ed47b classes/populate_sdk_ext: remove unnecessary dependencies breaking SDK_TARGETS
Up until recently it was possible to set SDK_TARGETS to include a native
recipe you wanted installed into the sysroot when installing the eSDK.
I'm not sure what happened but now when you try to add a native recipe
to SDK_TARGETS you get a missing task error because this recipe has no
do_package_write_* task. Of course such a task dependency is erroneous
and is apparently caused by setting SDK_RDEPENDS. I've checked and it
turns out that we no longer need to set SDK_RDEPENDS anyway (probably
because we explicitly set up task dependencies further down in the
class, which I don't think we were fully doing in early versions of the
eSDK). Thus, drop setting this variable to restore the functionality.

(From OE-Core rev: fe8268ede33a8c76492b85da0f41062f47a5279c)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-07 14:50:09 +00:00
Richard Purdie 529582a238 staging: Add missing parameter to debug message
(From OE-Core rev: 77ac40f8d4a4718b05f3692222a0dc91d90f0ab8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-07 14:50:09 +00:00
Richard Purdie ab78c782f7 staging: Improve file creation resiliance
If you abort a build mid execution of the staging extend_sysroot function
there are currently races and cleanup of that function may fail.

This change splits the code into separate phases so that the manifests are
manipulated before files are installed, meaning we should be able to reverse
actions if builds fail, crash or are interrupted.

(From OE-Core rev: fc504b6b8538453d2f058d7cc69be098f52f8091)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-07 14:50:09 +00:00
Richard Purdie 175a457032 staging: Add shared manifest support
(From OE-Core rev: 51834006476786f847b3ca60d5ed83a952fc64cb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-07 14:50:09 +00:00
Richard Purdie c8b280e8a4 staging: Don't put ptest files into the sysroot
This wastes space/time as the ptest files aren't going to be used in the
sysroot so add them to the blacklist.

(From OE-Core rev: 61591fe2eea588ff92ff7f81f2ed8844617f5307)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-07 14:50:09 +00:00
Ed Bartosh 791391e257 image_types_wic: remove dependency to do_bootimg
Removing task dependency do_wic -> do_bootimg as wic
doesn't depend on hddimg/booimg anymore.

(From OE-Core rev: af52d878940babb88bc8a2650e8c0115b6ccaf9f)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-05 09:22:18 +00:00
Ed Bartosh faa2823801 image: rename image-wic -> image_types_wic
Make name of the wic image type class consistent with
existing naming scheme for image types.

(From OE-Core rev: 4aab1b77d5f9403cbb3fae790069ef54821491fb)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-05 09:22:18 +00:00
Bruce Ashfield fcc40ac908 kernel-yocto: log the BSP definition file
When debugging a kernel configuration issue, one of the first questions
is "what BSP was used". To answer this qusetion, we log the BSP .scc
file that was used to generate the kernel configuration in the kernel
source meta directory.

(From OE-Core rev: 1003615b866bfe196994bdc25d305ff5eace2715)

Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-05 09:22:17 +00:00
Bruce Ashfield 80a74318de kern-tools: re-enable scc merge command
The ability to merge two branches directly from a .scc file was
dropped during the streamlining of the tools.

As was pointed out by David Vincent <freesilicon@gmail.com>, there is
once again a valid use case for this functionality, so we restore the
capability.

(From OE-Core rev: a0059ebbb52c659282e355664bba1a2fa282170e)

Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-05 09:22:17 +00:00
Richard Purdie f431f766a0 uninative: Make patchelf modified files sparse
When we switched to recipe specific sysroots (rss), performance took a nose dive. Its
easy to blame rss but it turns out not to be entirely at fault.

Three configurations are compared here:

a) Pre-RSS (revision 45df694a9f)
b) Post-RSS (revision 226a508da9)
c) as b) with this change

Overall build times:

a) 22794.25user 2687.88system 30:32.84elapsed 1390%CPU (0avgtext+0avgdata 919056maxresident)k
b) 22677.25user 3238.79system 36:16.68elapsed 1190%CPU (0avgtext+0avgdata 918896maxresident)k
c) 23571.84user 3383.65system 31:36.83elapsed 1421%CPU (0avgtext+0avgdata 919068maxresident)k

For the overall build and sstate directories, du -s shows:
a)
3992588   build-pre-rss/sstate-cache
30804484  build-pre-rss/tmp
b)
4013272   build-with-rss/sstate-cache
36519084  build-with-rss/tmp
c)
4014744   build-with-rss2/sstate-cache
35336960  build-with-rss2/tmp

However more worryingly:

$ du -s build-pre-rss/tmp/sysroots/
2506092 build-pre-rss/tmp/sysroots/
$ du -s build-with-rss/tmp/sysroots-components/
3790712 build-with-rss/tmp/sysroots-components/
$ du -s build-with-rss2/tmp/sysroots-components/
2467544 build-with-rss2/tmp/sysroots-components/

These numbers *should* be equivalent but as you can see, b) is ~1.2GB larger. The reason turned out
to be patchelf. Taking a specific binary from a specific recipe, bc from bc-native, in a) its 82kb
(stripped) yet in b) its 2.17MB.

$ ./patchelf --set-interpreter /bin/rp bc
warning: working around a Linux kernel bug by creating a hole of 2084864 bytes in ‘bc’

https://github.com/NixOS/patchelf/blob/master/src/patchelf.cc#L710 shows that this "hole" is just
padded zeros using memset, its not a proper sparse hole.

This patch copies files with cp --sparse=always after modifying them with patchelf, then replacing
the original file. The better fix will be to fix this in patchself itself and seek() there
when writing the new file but that means new uninative tarballs and will take a bit of work
so I'm proposing this workaround in the meantime.

Also, this patch drops error handling since subprocess check_output() tracebacks will print this
information if the command fails so we can simplify the code.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-02 17:58:24 +00:00
Richard Purdie 85e9af9d78 staging: Reduce the number of mkdirs calls
The number of mkdir calls was showing up high on the profile charts since
it was getting called once per file which is excessive. Each call results
in one or more syscalls which is bad for performance. Cache which
directories we've seen to reduce the calls to a more reasonable number
and speed up recipe specific sysroots.

(From OE-Core rev: 680fb343be5d0a7f9abbe9c75ca232abe5871663)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-01 18:02:12 +00:00
Richard Purdie 758f18167f sstate/staging: Batch log messages for performance
According to profile data, repeated calls to bb.debug and bb.note in
the extend_recipe_sysroot() codepath were accounting for 75% of the time
(1.5s) in calls from tasks like do_image_complete.

This batches up the log messages into one call into the logging system
which gives similar behaviour to disabling the logging but retains the
debug information.

Since setscene_depvalid is also called from bitbake's setscene code,
we have to be a little creative with the function parameters and leave
the other debug output mechanism in place. This should hopefully
speed up recipe specific sysroots.

(From OE-Core rev: 3b0af8dc0f796345d1f1ba77ea35bbd090a5feb3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-01 18:02:12 +00:00
Richard Purdie e9256edb33 relocatable: Make native .pc files relocatable
The native .pc files currently have hardcoded paths in them meaning each has
to be relocated at final install time. pkg-config has built in functionality
to avoid this, namely the pcfiledir variable.

This function translates .pc files to use the variable meaning further
relocation later is unnecessary.

(From OE-Core rev: a3f4e9ff55c7d9eba175e83eb6c99cf349a74432)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-31 23:47:33 +00:00
Richard Purdie 2bb0b1d9b5 utils: Avoid need to relocate wrapper scripts
Whilst the path to the executable is dynamically determined, the passed in
environment variables or parameters are not relocatable and rely on the sstate
remapping code. In the recipe specific sysroot case this has become more costly.

This patch translates such paths into relocatable entries which means that
a sed replacement at final installation isn't needed.

(From OE-Core rev: 3e02a7cda0ba70e3f9886c5cce37d71361280b11)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-31 23:47:33 +00:00
Aníbal Limón 90b4e075f4 oeqa/runtime/cases: Rename syslog module to oe_syslog
Debian based distros has a builtin syslog module so when
try to load tests using unittest it references the builtin
module instead of runtime/cases.

[YOCTO #10964]

(From OE-Core rev: 9923e3cdb58c2b3c54ec5fe99b2cec4cdc9fff92)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-31 23:27:47 +00:00
Mariano Lopez 3eef8f7955 testimage.bbclass: Fixes about suites and dependencies
There are some missing suites on sato-sdk and lsb-sdk images so add it.

The createrepo-native needs to be built before run testimage in
order to create a repository from install packages.

The DL_DIR data also needs to be updated from the current bitbake
environment instead of use the value from testdata file.

[YOCTO #10964]

(From OE-Core rev: 42f3026b39519ba98ae3153329622ce670d57c09)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-31 23:27:47 +00:00
Patrick Ohly a15895dac1 populate_sdk_ext: fix == bashism
Found via verify-bashisms.

(From OE-Core rev: b690b8dc634844b4f6672d524f0e6f1d86dc4e20)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-31 15:28:41 +00:00
Ming Liu 5707949f03 rootfs-postcommands.bbclass: drop insert_feed_uris function
The source makes no reference to insert_feed_uris any more, simply drop
the unused reference, to avoid confusing the developers.

(From OE-Core rev: 26dddb7d4a4945abaaefd460a342de6abd609594)

Signed-off-by: Ming Liu <peter.x.liu@external.atlascopco.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-31 14:43:02 +00:00
Ming Liu e2280b9e12 meta: remove remaining True option to getVar calls
This is a complementary fix to commit 7c552996:
[ meta: remove True option to getVar calls ]

it intended to remove all True option to getVar calls, but there are
still some remaining.

Search made with the following regex: getVar ?\((.*), True\)

(From OE-Core rev: 87d03ffe03d6f01e360bfd51714be96e62506e0a)

Signed-off-by: Ming Liu <peter.x.liu@external.atlascopco.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-31 14:43:02 +00:00
Mariano Lopez 1e88c8fce4 testimage.bbclass: Allow to run tests on autobuilder's images
With the change to the new framework data store dependecy was
removed, instead a new file is generated and used in testimage.
When testing builds from the autobuilders the test data values
are from the autobuilder, including the paths.

Some tests require paths to current environment in order to run,
this commit will update such paths and fix the error of running
images donwloaded from autobuilders.

[YOCTO #10964]

(From OE-Core rev: 26ad5105fc2ce03b7ee8ecc6911fd40a52bd573a)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-31 14:43:01 +00:00
Ola x Nilsson 2e77eff6cc externalsrc: Hide created symlinks with .git/info/exclude
Add created symlinks to the exclude file.  This will both make them
less distracting and hide them from the srctree_hash_files function.

(From OE-Core rev: c11fcd6fbde4a90913960b223451e0ce9e6b4b64)

Signed-off-by: Ola x Nilsson <olani@axis.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-31 14:43:01 +00:00
Kristian Amlie 2313a77d84 sstate: Make SSTATE_SCAN_CMD vars configurable using weak defaults.
For this we move them out of the python section so they can be
overridden on a per-recipe basis.

The motivation for this change is that not all tool chains need the
path modifications provided by the command, and these will provide
alternative or empty commands. The Go compiler is such an example.

(From OE-Core rev: 7d2a2160336413736dd4640f5b84ba4d74bb00f5)

Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-31 14:43:01 +00:00
Ed Bartosh 5302fb4620 toolchain-shar-extract: compare SDK and host gcc versions
If ext sdk is built by gcc version higher than host gcc version
and host gcc version is 4.8 or 4.9 the installation is known to
fail due to the way uninative sstate package is built.

It's a known issue and we don't have a way to fix it for above
mentioned combinations of build and host gcc versions.

Detected non-installable combinations of gcc versions and
print an installation error.

[YOCTO #10881]

(From OE-Core rev: bfb9ea2604557804bd8c16adb57ccdd868a4020f)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-31 14:43:01 +00:00
Ross Burton e286829fcc package_deb: handle allarch when mapping tune to dpkg architectures
It was discovered that buildtools-tarball can't be built with Debian packaging:

E: Unable to locate package nativesdk-buildtools-perl-dummy

The package exists but dpkg doesn't see it because the Architecture field isn't
valid: for this package the Architecture field is 'allarch' which isn't in the
list of valid architectures we define.

package_deb already has a mapping of OE architectures (PACKAGE_ARCH) to dpkg
architectures (DPKG_ARCH), for example our x86_64 is amd64 in dpkg.  However
allarch.bbclass sets an invalid TARGET_ARCH of 'allarch' (to break anything
which attempts to use the target architecture) and package_deb relied on the
PACKAGE_ARCH check later turning the Architecture field into 'all'.

This usually works, but nativesdk-buildtools-perl-dummy inherits allarch but
then changes PACKAGE_ARCH to buildtools-dummy-nativesdk to isolate the packages
in a separate feed.  We can handle this by explictly checking for allarch in the
TARGET_ARCH check, as that everything that inherits allarch will want to have
Architecture: all in their packages.

[ YOCTO #10700 ]

(From OE-Core rev: 58a5771f20f7851816b5f39df0ada8f16438065c)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-31 14:43:00 +00:00
Ross Burton 9533e14e6f distrodata: add DATETIME to vardepsexclude for do_distro_check.
This task uses DATETIME so add it to vardepsexclude so bitbake doesn't refuse to
execute the task when the time changes between the controller and the worker
calculating the hash.

[ YOCTO #10960 ]

(From OE-Core rev: e6784ef0d56a1b297001bc59c0da3aecebd1c656)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-31 14:43:00 +00:00
Ed Bartosh 75df73526a grub-efi.bbclass: use 'grub-efi-' prefix
grub-efi recipe added 'grub-efi-' prefix to the file name of
efi binary. Changed grub-efi.bbclass accordingly.

(From OE-Core rev: 860c44435ea7cbbe3517d5ab9146213cc80fd0e7)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-31 14:38:32 +00:00
Ed Bartosh 6740b8fdbb isoimage-isohybrid: use TRANSLATED_TARGET_ARCH instead of MACHINE_ARCH
isoimage-sihybrid plugin uses MACHINE_ARCH to get the name of initrd image.
It doesn't work for all machines, for example for quemux86-64 machine
MACHINE_ARCH is quemux86_64 and initrd name is
core-image-minimal-initramfs-qemux86-64.cpio.gz

Used TRANSLATED_TARGET_ARCH variable to get the initrd image name.
Replaced MACHINE_ARCH->TRANSLATED_TARGET_ARCH in WICVARS variable to
make it available from <image>.env file.

(From OE-Core rev: edf0830412de66eada9ef3f3947ca42e301c5377)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-31 14:38:32 +00:00
Ed Bartosh 62310b3726 image-wic: remove HDDDIR from WICVARS
Removed HDDDIR as it's not used by wic anymore.
Stopped usage of HDDDIR in wic test suite.

(From OE-Core rev: b3ed3a79852f85651c8fa192a8118f9904e88ef8)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-31 14:38:32 +00:00
Ed Bartosh a508cb384c wic: use INITRD_LIVE in isoimage-isohybrid
INITRD variable is not set if hddimg is disabled.
isoimage-isohybrid can't get correct name for initrd if INITRD
variable is not set.

Added INITRD_LIVE to WICVARS and used it in isoimage-isohybrid
code to get initrd artifact name. Used INITRD if INITRD_LIVE is not set.

(From OE-Core rev: c672753be60550b15d5d593c251b61776d5b104d)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-31 14:38:32 +00:00
Ed Bartosh 02dafba6ed image-wic: move wic code to image-wic.bbclass
There is a lot of wic code in image.bbclass and image_types.bbclass
Having all code separated in one file should make it more readable
and easier to maintain.

(From OE-Core rev: 786368568a9525212e69f5cbf6da236f0a6be013)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-31 14:38:32 +00:00
Ed Bartosh c01cc6ff24 wic: change location of .env files
Current location of .env files $STAGING_DIR/imagedata. It doesn't
depend on machine and be rewritten by the builds for different
machines.

Changed location to $STAGING_DIR/$MACHINE/imagedata to avoid .env
files to be rewritten.

(From OE-Core rev: 94245144f5cef344d90bc2a7b3267cdae9d192e4)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-31 14:38:32 +00:00
Ed Bartosh 6d78ff1144 image_types: use correct output directory
Wic put result images into <output dir>/build, which was confusing.
Now it's fixed in wic code and images are put into output directory.
Changed code in image_types to reflect this.

[YOCTO #10783]

(From OE-Core rev: 3983507bf859f6c3c29e9dea06ffe51f42c0759c)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-31 14:38:32 +00:00
Richard Purdie d45d4a5a21 staging: Handle stale sysroot rebuild issue
If a stale sysroot object was found the complete stamp wasn't removed,
meaning later code could fail when the new link creation was attempted.
Ensure the stale complete stamp is also removed if present.

(From OE-Core rev: d7d26488f252d60628862ee114a4404e8d6cb6f1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-28 23:32:16 +00:00
Richard Purdie f66f1af4b1 staging: Fix missing nativesdk do_populate_sysroot depends on binutils
Recent changes exposed the fact this dependency was missing in nativesdk
cases, fix this.

(From OE-Core rev: 46bd7e48cb5ae654d915fa493420534219682e85)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-28 23:32:16 +00:00
Richard Purdie 1d4c20ecc2 staging: Allow removal of stale sysroot objects
The main sysroot components of unreachable build targets will be removed by the core
code. This currently doesn't trigger a removal in the individual workdirs. This
adds in symlinking between the complete stamps and the component sysroot meaning
we can detect when someting was removed and hence remove it from the sysroot.

This fixes cases where DISTRO_FEATURES like systemd are changed amongst other
things and makes builds more robust against configuration changes.

If a dependency is rebuild, that is caught by checksum comparision code elsewhere
in this function as before.

(From OE-Core rev: 7656347c556915dc9acc5a2311ec4e879de6f43d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-28 23:32:16 +00:00
Richard Purdie a4274bcc37 staging: Drop common prefix WORKDIR from manifest files
Manifest files containing the same duplicated prefix are wasteful on space
and ultimately this costs build time. Drop the WORKDIR prefix from the manifest
files since this small change mounts up a lot.

(From OE-Core rev: 1cb245a99762e21a170b6a9beabb07e558424946)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-28 23:32:16 +00:00
Richard Purdie a90c8817a5 sstate: Allow sstate_clean_manifest to take a prefix
Manifest files containing the same duplicated prefix are wasteful on space
and ultimately this costs build time. Add support for manifest files with
common prefixes removed and use the prefix if the path isn't absolute.

(From OE-Core rev: 71e8c98238d627f9665faee9d7c226b9c07bd256)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-28 23:32:16 +00:00
Richard Purdie ab0f46400c gzip/pigz/expat/image: Simplify gzip-native/pigz-native
With recipe specific sysroots, the gzip-replacement-native dance/class
is obsolete, simplify the code accordingly.

(From OE-Core rev: 39865fdf3698a130f792d41853f9c9ca1901e335)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-28 23:32:16 +00:00
Richard Purdie e6e0ac4fb8 allarch: Drop STAGING_DIR_HOST expansion
Now that STAGINGDIR_HOST doesn't contain MACHINE, we no longer need to expand
the value. Pre-expansion can mean components like PV can be expanded too early
and cause problems for certain use cases.

(From OE-Core rev: d04f0363a1e8ae641da3e80ffbd4c8803cb1c91a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-28 23:32:16 +00:00
Richard Purdie 60662a2117 image/kernelsrc/packagegroups/recipes: Remove uneeded noexec tasks
We used to have issues removing tasks like do_fetch due to implications
for targets like world and universe. These have now been resolved.

Removing uneeded tasks has advantages compared to noexec since it means
that accidentally left in dependencies are no longer needed/processed
(e.g. do_patch depends on quilt-native).

This cleans up a number of cases which local analysis highlighted as
being unneeded leading to slightly cleaner task graphs.

(From OE-Core rev: 4e6ee37e09c60e83c0dfd844ba9cf8a07507f099)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-28 23:32:16 +00:00
Patrick Ohly 46ed698f05 image_types.bbclass: rebuild .wks file when .wks.in changes
WKS_FILE(S) can refer to .wks.in files which get expanded during the
build by do_write_wks_template. The actual content of the .wks.in file
gets added to the recipe meta data during parsing, and thus we need to
ensure that the recipe gets re-parsed when the file changes.

This fixes two related problems:
- editing the .wks.in file and rebuilding an image did not recreate
  the image unless something else changed or "bitbake -c clean" was
  used explicitly
- when forcing a rebuild, the cached meta data and the actual one
  do not match, leading to "ERROR: Taskhash mismatch ... for ....bb.do_write_wks_template"

(From OE-Core rev: d00b95b18237b276c221b16dfc511a6da150ef06)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-28 23:32:16 +00:00
Mariano Lopez a02f7f2a33 testimage.bbclass: Add systemd test automatically
This adds systemd test automatically when an image
is build with systemd DISTRO_FEATURE

[YOCTO #10935]

(From OE-Core rev: 2e6ba167bee7c7b42b1e0e96eb5a7be2eb0fd2c0)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-26 10:44:27 +00:00
Ioan-Adrian Ratiu 6805ae0f2a useradd_base: keep group if it still has users
perform_groupdel() tries to delete a group irrespective if other
users have it as their primary group, thus the call to groupdel will
fail with the following error error:

groupdel: cannot remove the primary group of user '<user>'

Add a check to perform_groupdel() to determine if there are other
users and keep the group, printing a warning. This is called right
after a user is deleted to delete it's group. If the last user is
deleted, only then the group is also deleted.

(From OE-Core rev: bba1f02cb6d80279c6fed34bb18f010aa921fc1e)

Signed-off-by: Ioan-Adrian Ratiu <adrian.ratiu@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-26 10:44:27 +00:00
David Vincent fdaa2d181d useradd-staticids: Fix groupadd
When a group name is duplicated between USERADD_PARAM and
GROUPADD_PARAM, this class tries to add it twice which results in
failure of the groupadd command.

(From OE-Core rev: 5b84cc895500c28674d2a7b7c2dd618cf8fb30e9)

Signed-off-by: David Vincent <freesilicon@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-26 10:44:27 +00:00
Joshua Lock 0eafcc05d8 toolchain-scripts: remove CCACHE_PATH from environment script
CCACHE_PATH was added to the toolchain-scripts in commit
1d31ddb856a80ba9da1a64ed5d701dc0f7351ef7 without a detailed
explanation as to why. This commit removes that environment
variable as it's causing failures since the introduction of
host_gcc_version() to oe.utils in
d36f41e5658bbbb6080ee833027879c119edf3e0, as used by
uninative to determine NATIVELSBSTRING.

This causes errors when the host has ccache available in in PATH
- host_gcc_version() fails because ccache has been told to only
look for the compiler in CCACHE_PATH and can't find gcc in order
to check its version. Toolchain compilers aren't detected due to
the binaries being prefixed with the triplet of the target.

(From OE-Core rev: e47fb403c2cb472d2666ed3ba4b10933467c8c7b)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-26 10:44:27 +00:00
Khem Raj 27c106e087 autotools.bbclass: Change acpaths defaults to be a weak assignment
We currently have to override the default setting using a particular
assignment order in the recipe, setting it weakly helps to get rid
of this problem.

(From OE-Core rev: 57be81156f74666ff2ed2755f5faa48edf00e4dc)

Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-26 10:44:27 +00:00
Richard Purdie da114c7d7d staging: Handle overlapping files for build-sysroot
The use of bb.fatal means build-sysroots fails as soon as multiple gdb's are
built with overlapping files, or multiple recipes with overlapping headers
exist.

Change the fatal call into an exception which we can then trap. Also
avoid trying to call readlink on something with isn't a symlink.

This allows build-sysroots to work better under various scenarios.

(From OE-Core rev: e20343a90e401bc92167867729076d321081d120)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-26 10:44:27 +00:00
Richard Purdie 7fb3c6a40a image/packagegroup/populate_sdk: Drop do_populate_sysroot task properly
Setting do_populate_sysroot as noexec means the code keeps thinking it can find
a manifest file for it. It also complicates sstate installtion since the code
would believe there is an sstate object there it should look for.

Instead, delete the task. This causes sdk failures as the dependencies are wrong
so fix those as well.

(From OE-Core rev: bd7d0314038a4c1a8e8c9ebdb7194f8e17db3fef)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-26 10:44:27 +00:00
Richard Purdie 1cec0f3bf4 cross: Update MULTIMACH_TARGET_SYS to reflect modern recipes
Mixing PACKAGE_ARCH with BUILD_* has always seemed somewhat odd. This
comes into play when building different SDKMACHINES as the PN space
for the recipes did once overlap. With the modern build process we
use, this overlap is a thing of the past since PN has SDK_SYS appended
to it.

cross.bbclass sets PACKAGE_ARCH == BUILD_ARCH so this change only affects
crosssdk.bbclass where PACKAGE_ARCH == SDK_ARCH.

The current structure is quite confusing and this removes this artefact
of history, allowing other gcc cleanups rather than meaning we futher
complicate the code.

(From OE-Core rev: 525bbfd08dacba84c92da0ab34310a85369526b7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-26 10:44:27 +00:00
Richard Purdie bb5fe4c35d staging/allarch: Remove hardcoded PACKAGE_ARCHS from the class
The code was making an assumption that the only PACKAGE_ARCH in use
was TUNE_PKGARCH. This is incorrect so iterate over the list from
PACKAGE_EXTRA_ARCH instead.

We also need to change allarch to preserve this variable, else the
staging code doesn't function. We do this in a way which clears the
variable history so that the task hashes remain unaffected.

[Thanks to Andrew Goodbody <andrew.goodbody@cambrionix.com> for
testing/fixing]

(From OE-Core rev: b1bab7a6eff6b195824be7d754de58c6e9ee2bfb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-26 10:44:27 +00:00
Richard Purdie 6ab4c5033a classes/oeqa: Replace subprocess.check_call() with check_output()
If you use subprocess.check_output() the traceback will contain the output
when the command fails which is very useful for debugging. There is no
good reason not to use this everywhere.

(From OE-Core rev: ad750dd1cc9d789abe723daddd098ce41d8547f5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 23:30:57 +00:00
Richard Purdie 3b4a36be50 sanity/abi-version: Force a clean TMPDIR after recipe specific sysroot changes
With the recipe specific sysroots, we need a clean tmpdir, else pseudo-native,
openssl-native, subversion-native and serf-native need to be manually cleaned.
After these there are probably more places where software doesn't rebuild correctly
even if we pass in new parameters to it.

The simplest solution is to force people to start from a clean TMPDIR since
everything would rebuild anyway.

(From OE-Core rev: da58e27a0f8fc8200f1953f05888834abd79c9f8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 23:30:14 +00:00
Rick Altherr 4aa6644f92 kernel-fitimage: Use compressed ramdisks in FIT images if available
kernel-fitimage:fitimage_assemble() was calling copy_initramfs from
kernel.bbclass which decompresses the initramfs cpio.  Assume that if
INITRAMFS_FSTYPES includes a compressed cpio, that is what it desired in
the FIT image.

(From OE-Core rev: 842ad404b36e00c89f615a3f7db4a2d30062effa)

Signed-off-by: Rick Altherr <raltherr@google.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:23 +00:00
Robert Yang be60ae52f2 qemuboot.bbclass: add blank lines in comments
Add blank lines in comments to make it easy for readind and updating.

(From OE-Core rev: 3f3344d1c063b0013a9ae7203bac30ab8f4ea17c)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:22 +00:00
Robert Yang 9dd223bf18 runqemu: fixes for slirp, network device and hostfwd
Fixed:
- Add QB_NETWORK_DEVICE to set network device, it will be used by both
  slirp and tap.
- Set QB_NETWORK_DEVICE to "-device virtio-net-pci" in qemuboot.bbclass
  but runqemu will default to "-device e1000" when QB_NETWORK_DEVICE is
  not set, this is because oe-core's qemu targets support
  virtio-net-pci, but the one outside of oe-core may not,
  "-device e1000" is more common.
- Set hostfwd by default: 2222 -> 22, 2323 -> 23, and it will choose a
  usable port when the one like 222 is being used. This can avoid
  conflicts when multilib slirp qemus are running. We can forward more
  ports by default if needed, and bsp.conf can custom it.
- Use different mac sections for slirp and tap to fix conflicts when
  running both of them on the same host.

[YOCTO #7887]

CC: Nathan Rossi <nathan@nathanrossi.com>
CC: Randy Witt <randy.e.witt@linux.intel.com>
(From OE-Core rev: 7dddd090806914a62d977730440d803e48f44763)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:22 +00:00
Mariano Lopez 46ce0b5ef3 testexport.bbclass: Migrate testexport to use new framework
This migrates current testexport implmentation to use the
new OEQA framework.

[YOCTO #10686]

(From OE-Core rev: 92cb884c989460563a063b29d2be8b7acd20577e)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:21 +00:00
Mariano Lopez 58789be270 testimage.bbclass: Add package install feature
This allows to use the package install feature with
the new OEQA framework.

[YOCTO #10234]

(From OE-Core rev: 077dc19445574457769eb4f231de97e8059cb75e)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:20 +00:00
Mariano Lopez f8d7db1905 testimage.bbclass: Add support for package extraction
testimage support the installation of packages without a package
manager in the target. This adds support for package extraction
required to support the installation feature.

[YOCTO #10234]

(From OE-Core rev: 8c7335290cb00ed0683241249297ca573ebd353a)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:20 +00:00
Mariano Lopez 41e4db0eea runtime/cases/smart.py: Migrate smart tests
This migrates the smart test from the old framework to
the new one. This has its own commit because smart
test was using bb and oe libraries that are available
when exporting the test cases to run in a different host.

Because of the removal of bb and oe libraries index and
packages feeds creation will be managed in testimage bbclass.

[YOCTO #10234]

(From OE-Core rev: 8d64ac4208e8dcb8a6fde6ea2959c9b3edfe2172)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:20 +00:00
Mariano Lopez 3857e5c91d testimage.bbclass: Migrate class to use new runtime framework
This migrates testimage class to use the new framework. Most of
the code added here is to get rid off the data store dependency.

[YOCTO #10234]

(From OE-Core rev: 2aa5a4954d7610f31875ba7e655f25f8892517b6)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:20 +00:00
Aníbal Limón 2d34b08533 oeqa/utils/dump: Move get_host_dumper to OERuntimeTestContextExecutor class
To avoid getVar calls inside a utils module, also moves
get_host_dumper import inside testexport isn't needed.

[YOCTO #10231]

(From OE-Core rev: f8beaf94e943a8b20d146be47a756af312ef107c)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:20 +00:00
Aníbal Limón ba1aec3407 oeqa: Fix files handling on runtime tests.
Common files was move to oeqa/files from oeqa/runtime/files
because the same files are used across Runtime,SDK,eSDK tests.

(From OE-Core rev: f099302efe8f222c3e4ae3604429f5ede4fd8c67)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:19 +00:00
Aníbal Limón e5f05409d0 oeqa/runtime: Fix TargetBuildProject instances
TargetBuildProject was refactored to avoid bitbake dependency so
the instance don't allow to pass data store anymore.

classes/testimage: Export proxies before run tests

The TargetBuildProject based tests download archives from network.

(From OE-Core rev: e275f29de500a338a02402ecc570405309963b35)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:19 +00:00
Aníbal Limón 03d175f1fa classes/testsdk: Migrate to use the new OESDKExtTestContext
(From OE-Core rev: b254822dad850ce74563c83b7a9e31463501baa7)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:19 +00:00
Aníbal Limón 3fac2417dd classes/testsdk: Remove the need of TEST_LOG_DIR variable
The TEST_LOG_DIR was used for store sdk_target_log this log
contains the output of the run of build commands now that information
could be found also on log.do_testsdk under WORKDIR.

The log will continue to store into SDK_DIR instead of TEST_LOG_DIR.

(From OE-Core rev: 1c9ba4b698bab916d42b58255692a7bf3d773bbc)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:19 +00:00
Aníbal Limón 92c57a5db7 classes/testsdk: Migrates testsdk.bbclass to use new OESDKTestContext
The functionality provided is the same with imporvements on code
reuse and better interfaces.

(From OE-Core rev: 7a1ae3149965b162fb2c71fc7067e07a7a189249)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:19 +00:00
Aníbal Limón ad2168365f classes/populate_sdk_base: Add write_sdk_test_data to postprocess
This function will generates testdata.json per SDK type.

[YOCTO #10231]

(From OE-Core rev: bff1b66c6b88fd2d99d14cecba27ee2f7462711f)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:19 +00:00
Aníbal Limón 60e5310f56 classes/rootfs-postcommands: Add write_image_test_data
This function will generates testdata.json by image type.

[YOCTO #10231]

(From OE-Core rev: 4c0061422df472395ca0caa76ef92adb860f1167)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:19 +00:00
Ed Bartosh dc4a612e60 image.bbclass: put .env files to ${STAGING_DIR}/imgdata/
As STAGING_DIR_TARGET started to point to a recipe specific
sysroot wic is not able to add .env files when .wks file refers
to multiple rootfs recipes.

Used STAGING_DIR instead of STAGING_DIR_TARGET to make the
directory with .env files the same for all recipes.

(From OE-Core rev: 3797cfd7473d3f9b7c0d999dcf9cd9608c8c7c6c)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:17 +00:00
Ed Bartosh d781747102 image_types: add RECIPE_SYSROOT_NATIVE to WICVARS
Added RECIPE_SYSROOT_NATIVE to the WICVARS for wic
to be able to access it when run from bitbake.

(From OE-Core rev: 3443ff47374c7e20ea54e838609b991f3f72810e)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:17 +00:00
Ed Bartosh 06a4500004 image_types.bbclass: build wic-tools
Added dependency do_image_wic -> wic-tools:do_build to ensure
that all required tools are ready to use by wic.

(From OE-Core rev: 9f547605d2bec282c5679fb35cd5b28a37c5a0d5)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:17 +00:00
Richard Purdie 9107d6ca14 Switch to Recipe Specific Sysroots
This patch is comparatively large and invasive. It does only do one thing, switching the
system to build using recipe specific sysroots and where changes could be isolated from it,
that has been done.

With the current single sysroot approach, its possible for software to find things which
aren't in their dependencies. This leads to a determinism problem and is a growing issue in
several of the market segments where OE makes sense. The way to solve this problem for OE is
to have seperate sysroots for each recipe and these will only contain the dependencies for
that recipe.

Its worth noting that this is not task specific sysroots and that OE's dependencies do vary
enormously by task. This did result in some implementation challenges. There is nothing stopping
the implementation of task specific sysroots at some later point based on this work but
that as deemed a bridge too far right now.

Implementation details:

* Rather than installing the sysroot artefacts into a combined sysroots, they are now placed in
  TMPDIR/sysroot-components/PACKAGE_ARCH/PN.

* WORKDIR/recipe-sysroot and WORKDIR/recipe-sysroot-native are built by hardlinking in files
  from the sysroot-component trees. These new directories are known as RECIPE_SYSROOT and
  RECIPE_SYSROOT_NATIVE.

* This construction is primarily done by a new do_prepare_recipe_sysroot task which runs
  before do_configure and consists of a call to the extend_recipe_sysroot function.

* Other tasks need things in the sysroot before/after this, e.g. do_patch needs quilt-native
  and do_package_write_deb needs dpkg-native. The code therefore inspects the dependencies
  for each task and adds extend_recipe_sysroot as a prefunc if it has populate_sysroot
  dependencies.

* We have to do a search/replace 'fixme' operation on the files installed into the sysroot to
  change hardcoded paths into the correct ones. We create a fixmepath file in the component
  directory which lists the files which need this operation.

* Some files have "postinstall" commands which need to run against them, e.g. gdk-pixbuf each
  time a new loader is added. These are handled by adding files in bindir with the name
  prefixed by "postinst-" and are run in each sysroot as its created if they're present.
  This did mean most sstate postinstalls have to be rewritten but there shouldn't be many of them.

* Since a recipe can have multiple tasks and these tasks can run against each other at the same
  time we have to have a lock when we perform write operations against the sysroot. We also have
  to maintain manifests of what we install against a task checksum of the dependency. If the
  checksum changes, we remove its files and then add the new ones.

* The autotools logic for filtering the view of m4 files is no longer needed (and was the model
  for the way extend_recipe_sysroot works).

* For autotools, we used to build a combined m4 macros directory which had both the native and
  target m4 files. We can no longer do this so we use the target sysroot as the default and add
  the native sysroot as an extra backup include path. If we don't do this, we'd have to build
  target pkg-config before we could built anything using pkg-config for example (ditto gettext).
  Such dependencies would be painful so we haven't required that.

* PKDDATA_DIR was moved out the sysroot and works as before using sstate to build a hybrid copy
  for each machine. The paths therefore changed, the behaviour did not.

* The ccache class had to be reworked to function with rss.

* The TCBOOTSTRAP sysroot for compiler bootstrap is no longer needed but the -initial data
  does have to be filtered out from the main recipe sysroots. Putting "-initial" in a normal
  recipe name therefore remains a bad idea.

* The logic in insane needed tweaks to deal with the new path layout, as did the debug source
  file extraction code in package.bbclass.

* The logic in sstate.bbclass had to be rewritten since it previously only performed search and
  replace on extracted sstate and we now need this to happen even if the compiled path was
  "correct". This in theory could cause a mild performance issue but since the sysroot data
  was the main data that needed this and we'd have to do it there regardless with rss, I've opted
  just to change the way the class for everything. The built output used to build the sstate output
  is now retained and installed rather than deleted.

* The search and replace logic used in sstate objects also seemed weak/incorrect and didn't hold
  up against testing. This has been rewritten too. There are some assumptions made about paths, we
  save the 'proper' search and replace operations to fixmepath.cmd but then ignore this. What is
  here works but is a little hardcoded and an area for future improvement.

* In order to work with eSDK we need a way to build something that looks like the old style sysroot.
  "bitbake build-sysroots" will construct such a sysroot based on everything in the components
  directory that matches the current MACHINE. It will allow transition of external tools and can
  built target or native variants or both. It also supports a clean task. I'd suggest not relying on
  this for anything other than transitional purposes though. To see XXX in that sysroot, you'd have
  to have built that in a previous bitbake invocation.

* pseudo is run out of its components directory. This is fine as its statically linked.

* The hacks for wayland to see allarch dependencies in the multilib case are no longer needed
  and can be dropped.

* wic needed more extensive changes to work with rss and the fixes are in a separate commit series

* Various oe-selftest tweaks were needed since tests did assume the location to binaries and the
  combined sysroot in several cases.

* Most missing dependencies this work found have been sent out as separate patches as they were found
  but a few tweaks are still included here.

* A late addition is that extend_recipe_sysroot became multilib aware and able to populate multilib
  sysroots. I had hoped not to have to add that complexity but the meta-environment recipe forced my
  hand. That implementation can probably be neater but this is on the list of things to cleanup later
  at this point.

In summary, the impact people will likely see after this change:

* Recipes may fail with missing dependencies, particularly native tools like gettext-native,
  glib-2.0-native and libxml2.0-native. Some hosts have these installed and will mask these errors

* Any recipe/class using SSTATEPOSTINSTFUNCS will need that code rewriting into a postinst

* There was a separate patch series dealing with roots postinst native dependency issues. Any postinst
  which expects native tools at rootfs time will need to mark that dependency with PACKAGE_WRITE_DEPS.

There could well be other issues. This has been tested repeatedly against our autobuilders and oe-selftest
and issues found have been fixed. We believe at least OE-Core is in good shape but that doesn't mean
we've found all the issues.

Also, the logging is a bit chatty at the moment. It does help if something goes wrong and goes to the
task logfiles, not the console so I've intentionally left this like that for now. We can turn it down
easily enough in due course.

(From OE-Core rev: 809746f56df4b91af014bf6a3f28997d6698ac78)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:17 +00:00
Richard Purdie 0ccdfb116e package_rpm: Clean up pointless exception handling
The exception handling in this function seemed mildly crazy. Python will
given perfectly good or in several cases better information if we let its
standard traceback/exception handling happen. Remove the pointless code.

(From OE-Core rev: 5ecdab6c2589a83bbbc522074052ff4438782102)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-22 09:05:39 +00:00
Richard Purdie 2628a65ffe package_ipk: Clean up pointless exception handling
The exception handling in this function seemed mildly crazy. Python will
given perfectly good or in several cases better information if we let its
standard traceback/exception handling happen. Remove the pointless code.

(From OE-Core rev: 61390438aec4a1f9beb4d332821cc6cda82e0379)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-22 09:05:39 +00:00
Richard Purdie 596c9eff21 package_ipk: Improve failure debug output
Currently if the dpkg-deb command fails you see an error message like this:

ERROR: nss-3.27.1-r0 do_package_write_ipk: opkg-build execution failed
ERROR: nss-3.27.1-r0 do_package_write_ipk: Function failed: do_package_ipk

which is pretty much useless. If we use subprocess.check_output, we see a
traceback and then:

Exception: subprocess.CalledProcessError: Command '<cmd>' returned non-zero exit status 1

Subprocess output:
<output>

which is much easier to debug from.

(From OE-Core rev: 64c8366a805e9cf0168ea2331c50c8d6a70c6dc4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-22 09:05:39 +00:00
Richard Purdie fdb52ade90 package_deb: Clean up pointless exception handling
The exception handling in this function seemed mildly crazy. Python will
given perfectly good or in several cases better information if we let its
standard traceback/exception handling happen. Remove the pointless code
along with the duplicated key checking which was broken in the inner loop
by usage of the wrong variable.

(From OE-Core rev: f755b07b528e828618141eda402399d791efba4a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-22 09:05:39 +00:00
Richard Purdie 53a3c58774 package_deb: Improve failure debug output
Currently if the dpkg-deb command fails you see an error message like this:

ERROR: nss-3.27.1-r0 do_package_write_deb: dpkg-deb execution failed
ERROR: nss-3.27.1-r0 do_package_write_deb: Function failed: do_package_deb

which is pretty much useless. If we use subprocess.check_output, we see a
traceback and then:

Exception: subprocess.CalledProcessError: Command '<cmd>' returned non-zero exit status 1

Subprocess output:
<output>

which is much easier to debug from.

(From OE-Core rev: 18160442869f56ee71538bc2dc60d7cb6c08c8a2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-22 09:05:39 +00:00
Richard Purdie d6b3f722c7 populate_sdk_ext: Add wic-tools to BB_SETSCENE_ENFORCE_WHITELIST
wic-tools has tasks which would always rerun and not come from sstate
to ensure we have a correctly populated sysroot. This is low overhead
and can be ignored from an eSDK perspective.

(From OE-Core rev: b631fe91b3939b7981dbf62cb2fb70f7baebe403)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-22 09:05:39 +00:00
Richard Purdie 60d8ca3e96 Revert "populate_sdk_ext: whitelist do_package tasks"
Since Paul reverted the sstate.bbclass change which was checking the sstate
mirror test results, this change should also not be needed anymore.

This reverts commit e30f5002c4.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-22 09:05:39 +00:00
Richard Purdie a49ee524a9 utils: Factor out common multilib datastore creation function
This code is already used in two places and we need it in others so
turn it into its own function.

(From OE-Core rev: 2a57e1334ff261a7ab45084be0f217e2acfe99af)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-22 09:05:39 +00:00
Jussi Kukkonen fc7394fe2d kernel.bbclass: Add PACKAGE_WRITE_DEPS for postinst
The depmodwrapper dependency is not actually used by the class but
anyone using pkg_postinst_kernel-base() will need it.

(From OE-Core rev: e3f5290d8deba9b7cead73b52ac45a37228fece9)

Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-20 11:53:51 +00:00
Jussi Kukkonen b7d5e29dd7 fontcache.bbclass: Add PACKAGE_WRITE_DEPS for postinst
Qemu is used to run fc-cache on postinstall.

(From OE-Core rev: bd20404a78b258bb95df41066f844e08a21e53fa)

Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-20 11:53:51 +00:00
Jussi Kukkonen c8a488768b gconf.bbclass: Add PACKAGE_WRITE_DEPS for postinst
gconftool-2 is used during postinstall.

(From OE-Core rev: d4272e712fe26689f350142426c9893ffe088ddb)

Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-20 11:53:51 +00:00
Jussi Kukkonen 136e4781b3 gio-module-cache: Add PACKAGE_WRITE_DEPS for postinst
Qemu is used to run gio-querymodules on postinstall.

(From OE-Core rev: d5361cc25fb7dd749e69063be48275c52462f72c)

Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-20 11:53:51 +00:00
Jussi Kukkonen 1ceb9934b3 gsettings: Add PACKAGE_WRITE_DEPS for postinst
glib-compile-schemas is needed during postinstall.

(From OE-Core rev: e049427e09b51c09b55d7f1299a7b878fe21768a)

Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-20 11:53:51 +00:00
Jussi Kukkonen ef05d22456 gtk-immodules-cache: Add PACKAGE_WRITE_DEPS for postinst
Qemu is used to run gtk-query-immodules-* on postinstall.

(From OE-Core rev: 7103447b198a12a30fdee3f789ff9e0d81534d54)

Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-20 11:53:51 +00:00
Jussi Kukkonen c197a98cee mime.bbclass: Add PACKAGE_WRITE_DEPS for postinst
update-mime-database is used in postinstall.

(From OE-Core rev: 57d63e8e89980c53f40e12ca670fc3f58f7e4c0f)

Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-20 11:53:51 +00:00
Jussi Kukkonen 777ed33f44 update-alternatives.bbclass: Add PACKAGE_WRITE_DEPS for postinst
(From OE-Core rev: 00e1e87d0dcf13f3b30682f1fb066f7fa4eed205)

Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-20 11:53:50 +00:00