This patch is comparatively large and invasive. It does only do one thing, switching the
system to build using recipe specific sysroots and where changes could be isolated from it,
that has been done.
With the current single sysroot approach, its possible for software to find things which
aren't in their dependencies. This leads to a determinism problem and is a growing issue in
several of the market segments where OE makes sense. The way to solve this problem for OE is
to have seperate sysroots for each recipe and these will only contain the dependencies for
that recipe.
Its worth noting that this is not task specific sysroots and that OE's dependencies do vary
enormously by task. This did result in some implementation challenges. There is nothing stopping
the implementation of task specific sysroots at some later point based on this work but
that as deemed a bridge too far right now.
Implementation details:
* Rather than installing the sysroot artefacts into a combined sysroots, they are now placed in
TMPDIR/sysroot-components/PACKAGE_ARCH/PN.
* WORKDIR/recipe-sysroot and WORKDIR/recipe-sysroot-native are built by hardlinking in files
from the sysroot-component trees. These new directories are known as RECIPE_SYSROOT and
RECIPE_SYSROOT_NATIVE.
* This construction is primarily done by a new do_prepare_recipe_sysroot task which runs
before do_configure and consists of a call to the extend_recipe_sysroot function.
* Other tasks need things in the sysroot before/after this, e.g. do_patch needs quilt-native
and do_package_write_deb needs dpkg-native. The code therefore inspects the dependencies
for each task and adds extend_recipe_sysroot as a prefunc if it has populate_sysroot
dependencies.
* We have to do a search/replace 'fixme' operation on the files installed into the sysroot to
change hardcoded paths into the correct ones. We create a fixmepath file in the component
directory which lists the files which need this operation.
* Some files have "postinstall" commands which need to run against them, e.g. gdk-pixbuf each
time a new loader is added. These are handled by adding files in bindir with the name
prefixed by "postinst-" and are run in each sysroot as its created if they're present.
This did mean most sstate postinstalls have to be rewritten but there shouldn't be many of them.
* Since a recipe can have multiple tasks and these tasks can run against each other at the same
time we have to have a lock when we perform write operations against the sysroot. We also have
to maintain manifests of what we install against a task checksum of the dependency. If the
checksum changes, we remove its files and then add the new ones.
* The autotools logic for filtering the view of m4 files is no longer needed (and was the model
for the way extend_recipe_sysroot works).
* For autotools, we used to build a combined m4 macros directory which had both the native and
target m4 files. We can no longer do this so we use the target sysroot as the default and add
the native sysroot as an extra backup include path. If we don't do this, we'd have to build
target pkg-config before we could built anything using pkg-config for example (ditto gettext).
Such dependencies would be painful so we haven't required that.
* PKDDATA_DIR was moved out the sysroot and works as before using sstate to build a hybrid copy
for each machine. The paths therefore changed, the behaviour did not.
* The ccache class had to be reworked to function with rss.
* The TCBOOTSTRAP sysroot for compiler bootstrap is no longer needed but the -initial data
does have to be filtered out from the main recipe sysroots. Putting "-initial" in a normal
recipe name therefore remains a bad idea.
* The logic in insane needed tweaks to deal with the new path layout, as did the debug source
file extraction code in package.bbclass.
* The logic in sstate.bbclass had to be rewritten since it previously only performed search and
replace on extracted sstate and we now need this to happen even if the compiled path was
"correct". This in theory could cause a mild performance issue but since the sysroot data
was the main data that needed this and we'd have to do it there regardless with rss, I've opted
just to change the way the class for everything. The built output used to build the sstate output
is now retained and installed rather than deleted.
* The search and replace logic used in sstate objects also seemed weak/incorrect and didn't hold
up against testing. This has been rewritten too. There are some assumptions made about paths, we
save the 'proper' search and replace operations to fixmepath.cmd but then ignore this. What is
here works but is a little hardcoded and an area for future improvement.
* In order to work with eSDK we need a way to build something that looks like the old style sysroot.
"bitbake build-sysroots" will construct such a sysroot based on everything in the components
directory that matches the current MACHINE. It will allow transition of external tools and can
built target or native variants or both. It also supports a clean task. I'd suggest not relying on
this for anything other than transitional purposes though. To see XXX in that sysroot, you'd have
to have built that in a previous bitbake invocation.
* pseudo is run out of its components directory. This is fine as its statically linked.
* The hacks for wayland to see allarch dependencies in the multilib case are no longer needed
and can be dropped.
* wic needed more extensive changes to work with rss and the fixes are in a separate commit series
* Various oe-selftest tweaks were needed since tests did assume the location to binaries and the
combined sysroot in several cases.
* Most missing dependencies this work found have been sent out as separate patches as they were found
but a few tweaks are still included here.
* A late addition is that extend_recipe_sysroot became multilib aware and able to populate multilib
sysroots. I had hoped not to have to add that complexity but the meta-environment recipe forced my
hand. That implementation can probably be neater but this is on the list of things to cleanup later
at this point.
In summary, the impact people will likely see after this change:
* Recipes may fail with missing dependencies, particularly native tools like gettext-native,
glib-2.0-native and libxml2.0-native. Some hosts have these installed and will mask these errors
* Any recipe/class using SSTATEPOSTINSTFUNCS will need that code rewriting into a postinst
* There was a separate patch series dealing with roots postinst native dependency issues. Any postinst
which expects native tools at rootfs time will need to mark that dependency with PACKAGE_WRITE_DEPS.
There could well be other issues. This has been tested repeatedly against our autobuilders and oe-selftest
and issues found have been fixed. We believe at least OE-Core is in good shape but that doesn't mean
we've found all the issues.
Also, the logging is a bit chatty at the moment. It does help if something goes wrong and goes to the
task logfiles, not the console so I've intentionally left this like that for now. We can turn it down
easily enough in due course.
(From OE-Core rev: 809746f56df4b91af014bf6a3f28997d6698ac78)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There were a few straggling expansion parameter removals left for
getVar/getVarFlag where the odd whitespace meant they were missed
on previous passes. There were also some plain broken ussages such
as:
d.getVar('ALTERNATIVE_TARGET', old_name, True)
path = d.getVar('PATH', d, True)
d.getVar('IMAGE_ROOTFS', 'True')
which I've corrected (they happend to work by luck).
(From OE-Core rev: 688f7a64917a5ce5cbe12f8e5da4d47e265d240f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.
Search made with the following regex: getVar ?\(( ?[^,()]*), True\)
(From OE-Core rev: 7c552996597faaee2fbee185b250c0ee30ea3b5f)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When enabling multilib.conf, the world was rebuilding due to changes in the
pkg-config search path. This doesn't matter so exclude it from the checksums.
(From OE-Core rev: 22001ba163e80b114212580279339acd15fa7298)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sysroot paths (which happen to contain MACHINE) should not cause the signatures
to change every time MACHINE changes so exclude this from them.
(From OE-Core rev: 5f3773e84332e9100b8739adf4831269329e0033)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a multilib package depends on an allarch recipe that installs an
architecture-independent .pc file it will not be able to find the .pc file as
the recipe gets installed into the MACHINE sysroot but pkg-config looks in the
MLPREFIX-prefixed sysroot.
Solve this by extending PKG_CONFIG_PATH in multilib environments to include the
architecture-independent path in the MACHINE sysroot
(sysroots/MACHINE/usr/share/pkgconfig/).
(From OE-Core rev: 6a6dd4eb53dbf8a0c31ac2d68af5befebe727596)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is not entirely obvious that all reasonable configurations
will have multilib.conf strictly before the file which might
want to set MULTILIB_GLOBAL_VARIANTS. The x86-ish values here
look like reasonable default guesses, but shouldn't override
an explicit setting.
(From OE-Core rev: 2645c07892abe79f73379b1ca4196f593cb3b330)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we have a hierarchy of pkgdata directories and the code has to put together
a search path and look through each in turn until it finds the data it needs.
This has lead to a number of hardcoded paths and file globing which
is unpredictable and undesirable. Worse, certain tricks that should be
easy like a GL specific package architecture become problematic with the
curretn search paths.
With the modern sstate code, we can do better and construct a single pkgdata
directory for each machine in just the same way as we do for the sysroot. This
is already tried and well tested. With such a single directory, all the code that
iterated through multiple pkgdata directories and simply be removed and give
a significant simplification of the code. Even existing build directories adapt
to the change well since the package contents doesn't change, just the location
they're installed to and the stamp for them.
The only complication is the we need a different shlibs directory for each
multilib. These are only used by package.bbclass and the simple fix is to
add MLPREFIX to the shlib directory name. This means the multilib packages will
repackage and the sstate checksum will change but an existing build directory
will adapt to the changes safely.
It is close to release however I believe the benefits this patch give us
are worth consideration for inclusion and give us more options for dealing
with problems like the GL one. It also sets the ground work well for
shlibs improvements in 1.6.
(From OE-Core rev: 1b8e4abd2d9c0901d38d89d0f944fe1ffd019379)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Multilib with opkg hasn't worked since the --force-overwrite option was dropped in
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=88a9f6db3482623c9cb54fb03db1045051cec9f6
Since we merged the libexecdir changes, we also need the --force-maintainer flag
to avoid conf file conflicts.
Both these changes are suboptimal however the alternative is completely broken and
these changes only affect people who have multilibs enabled and use opkg.
[YOCTO #3453]
(From OE-Core rev: 6310e66cd62aab6109027a8dce9c56bf721bea92)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- skip the non-packagegroup allarch recipes in multilib_virtclass_handler
- extend PROVIDES/RPROVIDES for allarch recipes which are not packagegroups
- use variants from MULTILIB_GLOBAL_VARIANTS (lib32 lib64 libx32) to create
additional pkgdata files for multilib allarch: ${pkgdatadir}/${variant}-${PN}
and ${pkgdatadir}/runtime/${variant}-${pkg}
- use variants from MULTILIB_VARIANTS to create additional pkgdata files
for multilib kernel/module-base recipes
- add a sanity check to determine if the current multilib is in
MULTILIB_GLOBAL_VARIANTS
[YOCTO #2918]
[YOCTO #3440]
[YOCTO #3565]
[YOCTO #3568]
(From OE-Core rev: bc4da2573dfb59ea2fc4af359701818df20f7663)
Signed-off-by: Constantin Musca <constantinx.musca@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the STAGING_DIR_KERNEL is set in the multilib.conf, then it may be
set incorrected. The evaluation happens before TMPDIR and LIBC are
defined in other components.
Moving the definition process to the multilib.bbclass ensures that
everything has been loaded before it is set.
(From OE-Core rev: 6bd87edc383b40e300b0ef4bf851c39b698305cd)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
RP: The list of recipes in multilib.conf needs to go away and we need
to just be able to extend all recipes with the multilib class.
Tested by building and running lib32-core-image-sato-sdk.
[YOCTO #1563]
(From OE-Core rev: 5d691f1bc95dfb9109b07827b33957b1fa5b2fa4)
Signed-off-by: Bogdan Marinescu <bogdan.a.marinescu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
1. STAGING_KERNEL_DIR was getting an invalid MLPREFIX. Fix this by forcing the expansion in multilib.conf.
2. Added multilib rule for 'perf'.
(From OE-Core rev: 71907775f601f0a48b995cc3277b61645d2e5068)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This project has been unmaintained for some time, and even the OpenMoko
project is not using it any more (in favour of FSO). Since we have ofono
in OE-Core which replaces and surpasses its functionality, we can remove
libgsmd.
(From OE-Core rev: 4cd880c61e9d74dbf1a747f3654239cadadf45ce)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add the multilib support for this package to multilib.conf because error will
appear when building an lib32-core-image-lsb without this patch.
[YOCTO #2571]
(From OE-Core rev: 264425b0a54806c08c218bd8b500c20b8c3152f4)
Signed-off-by: Xiaofeng Yan <xiaofeng.yan@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
task-core-tools-debug, task-core-tools-profile, task-core-tools-testapps
otherwise if we choose one through PACKAGE_GROUPS all packages
are built since they are in same recipe.
(From OE-Core rev: 180227d8ff71c42e0d16ad7eb4a5f9b7d6a0c1c6)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds in various missing dependencies to ensure the set
of recipes listed for multilib support can be cleanly built.
(From OE-Core rev: 1d2697e6f00cc3771f39f5a7d4384a22d9696b16)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Enable these two core dependencies for multilib builds, fixing
multilib build failures that were occurring. I've checked and the
recipes are multilib capable.
[YOCTO #1835]
(From OE-Core rev: db446421933d640528328e46a29f828a36d6649b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It's very old at this point and doesn't build with recent clutter
(From OE-Core rev: 04d2bb8e83aaac856f03f3fc9772285bd82182e4)
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The D-Bus backend has been integrated into upstream GConf so we can
switch to upstream and drop gconf-dbus.
I've gone for a release in the 3.2 series as we disable Gtk+, and
therefore are not impacted by the gtk+3 changes, and the D-Bus backend
was unstable before this release.
(From OE-Core rev: f952b1bc8a4f307e2e2f3941f50becd72d88421d)
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* xserver-xorg is closer to upstream naming and
that's how it's named in OE-classic and meta-oe? It would make meta-oe
transition easier and better to do it now then convert meta-oe to
xserver-xf86 and then rename it back later.
(From OE-Core rev: 0b31c7200a368533df970f0efeb81e2e20c73593)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
currently we have allarch type of recipes, which may still have
architecture dependency, like x11-common. So we need to drop the
handling to allarch in multilib case.
Also remove the PV postfix in python-pygobject DEPENDS, since multilib
code will treat a native package multilib capable.
[YOCTO #1497]
[YOCTO #1498]
(From OE-Core rev: 64c0279e6b0d2325a326058476228360898550f3)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
[YOCTO #1481]
Since tasks are referred to almost exclusively via RDEPENDS, and bitbake
will build an entire task recipe when only one of its task packages are
actually needed, building a console-only image that uses
task-core-apps-console (or less directly, has apps-console in its
IMAGE_FEATURES) will cause a build of a whole list of X11 applications
that aren't needed. Splitting the task-core recipe into X11 and console
portions prevents this issue.
(From OE-Core rev: 07407baafb728c5ae1bb283f5b02a2ac773360bc)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Recipes like update-rc.d and qemu-config inherit "allarch", thus we
shouldn't add multilib BBCLASSEXTEND for them in multilib.conf.
Besides, we need to add multilib packages as the RPROVIDER contents for
those recipes, in order to avoid the NoProvider error when parsing.
[YOCTO #1471]
(From OE-Core rev: 329d864f9bbf94ad3aae8df43d63fe10e4237e4f)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Kernel should be unique in a multilib system, so we remove the
linux-yocto's extension in multilib.conf
(From OE-Core rev: db78d1289705a38f0069f84101296e9309d67fa3)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For a given system we only want one kernel to be built. This change makes
the main kernel recipe provide all of the provides of the various enabled
multilibs hence allowing it to fulfil all the appropriate dependencies.
To make this work a global multilib class file needed to be created.
This patch also enables this multi provider functionality for "allarch"
packages.
[YOCTO #1361]
(From OE-Core rev: 2fd257f6c610624f05c8dd3fe1486364af04696f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
libsdl is required by sato image, so extend it for multilib.
(From OE-Core rev: 88abab9ba9632e87f3a081915a69cbc5c36b4d3c)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Thinking of the senario that, if we already built out a 64bit image
along with the full toolchain bootstrapped, then we need to build some
32bit libraries, which needs lib32 versions of gcc and eglibc. These
toolchain recipes will bootstrap again in the same sysroot, resulting
that lib32-gcc-cross-initial will find some macros owned by eglibc have
already been defined and thus it includes non-existed headers that
provided by later lib32-eglibc.
The solution for the above issue is to use different sysroot for
multilib recipes, here we add ${MLPREFIX} in front of the machine
specific sysroot directory name.
[YOCTO #1372]
(From OE-Core rev: a1508ad1aec2d2f9ee040aa217c33193cd5bd871)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
1. Added variable MULTILIB_VARIANTS to store all the instance variants
for multilib extend.
2. Added function all_multilib_tune_values to collect the variable
values for all multilib instance.
3. multilib bbclass handler will save the orignal value of all variables
defined in MULTILIB_SAVE_VARNAME.
(From OE-Core rev: 18bba910e04bff75460f408e4557d4bae21ad592)
Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie
Joshua Lock
Ross Burton
Peter Seebach
Constantin Musca
Mark Hatle
Bogdan Marinescu
Paul Eggleton
Valentin Popa
Xiaofeng Yan
Lianhao Lu
Saul Wold
Dongxiao Xu
Khem Raj
Joshua Lock
Martin Jansa