In order to make sharing sstate files easier chmod them with 0664 permissions so that they are readable and writable by the user and any other members of their group.
(From OE-Core rev: dae9ad8a0ba0343e3083694cdcb20f0d02927ad0)
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fixmepath file that is generated contains a list of all of the files
that need their paths fixed. In the previous version the fixmepath was
generated to include all of the files that sed may have changed. In the
new version, we first grep the files to see if they contain a path that
needs to be changed, only then do we perform the sed operation on those files.
This results in a modest performance increate in the creation of the sstate
file. The following numbers include the do_package and do_populate_sysroot
tasks on the perl recipe.
Before the change:
real 4m23.018s
user 1m57.067s
sys 1m33.327s
After the change:
real 4m13.083s
user 1m54.062s
sys 1m26.064s
However, a more significnt performance gain is felt during the
extraction/install of sstate cache files, as the fixmepaths file now has a
significantly smaller list of files to modify.
Before the change:
real 0m39.798s
user 0m11.158s
sys 0m12.642s
After the change:
real 0m25.511s
user 0m8.408s
sys 0m5.077s
(All numbers above were recorded with a cold filesystem cache on a machine
with 12 GB of ram.)
(From OE-Core rev: 46067264bedeff8248a2b2441733420fe6651f84)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In a pathological case, lots of files to process, the sstate_installpkg
performance was very poor. It interated over each file and ran 3
individual sed commands per file. Changing this to keep iterating
but running only a single command took about 1/3 time time.
However, when looking at the corresponding sstate_hardcode_path
function, it was clear we could optimize this further.
Using the same encoding logic to specify only the minimumal sed
operation necessary, and using xargs to avoid the os.system call the
install step was able to be performed in 13% of the original time.
Example timing numbers for perl:
3m7s original code
1m20s single sed, but interating
0m26s using xargs and limited sed
(From OE-Core rev: d9f655753fbdc8cbd8e705577430fed4f23732b3)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Reluctantly revert this since it breaks the tar-native workaround we have
for old versions of tar :(
This reverts commit 01218e29f9.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is currently a race window when creating sstate packages since we don't
atomically write the files to SSTATE_DIR. This change ensures we do so by writing
to a temporary file and then doing an atomic move.
(From OE-Core rev: 52bf113e786a57123a9da98f64442afbc2f1471e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This resolves issues related to pigz-native when installing from sstate that people
have been seeing. It also gives us a way to solve issues like the gzip-native race
during sstate package creation covered in Yocto #1774.
(From OE-Core rev: 518dffe20178f5969dddccb17d6ab347afb72beb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* also fixes replacing paths for perl where cmd line was probably
too long for os.system(cmd) (it had 560410 characters because a lot of
files from sstate_scan_cmd).
* also print those 2 commands so we can find them in log.do_package
(From OE-Core rev: 94c52d68fc2ce258bcc5b0978ac73413480a1a93)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* e.g. mysql5 has mysql_config not mysql-config
(From OE-Core rev: aebdffc93ba8446bbd4e20263f0f75d4d4460c47)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The sstate_hardcode_path() function triggered large numbers of exec()
calls when processing packages with large numbers of file relocations
(e.g. perl). This patch optimises those calls into longer single commands
which make the code significantly more efficient.
This reduced the do_package time for perl by 2 minutes (from 4.75 minutes)
for me.
(From OE-Core rev: 4159ff761cf29a03aeb56c7686a4e7af02b52219)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We process all files in the native/cross cases for finding and
fixing relocation issues. In the target case we've only processed
.la and binconfig files. Since there are other files which are
in need of this processing, this change allows recipes to specify
files that may be outside the normal set. This means hardcoded
paths that need to be fixmepathed to work correctly are handled
and addresses some sstate relocation bugs that have been seen.
Based on a patch from Saul Wold <sgw@linux.intel.com>
(From OE-Core rev: 6ffdcd9120b572fa41659029c3bda7bf00ebcb77)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Complete the bb.data.getVar/setVar replacements with accesses
directly to the data store object.
(From OE-Core rev: 2864ff6a4b3c3f9b3bbb6d2597243cc5d3715939)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this change we can end up looking for <stamp>.${MACHINE}
instead of the expected expanded value.
(From OE-Core rev: 9f743b5033177216fe0e1d3e43ba831f356df08e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
sstate was being a little too ethusiastic about removing stamp files and
was removing stamp files for other machines when it shouldn't have been.
This patch teaches sstate about machine specific stamp extensions and
allows it to only remove the current task's stampfiles.
Based on a patch from Phil Blundell <philb@gnu.org> with some tweaks
from me.
(From OE-Core rev: 5e9488495401399d39fcb5012b86c313b6caca73)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If several sstate packages are decompressing at the same time, they can
conflict over the "fixmepath". If two packages try to write to this at
the same time it results in an error. To avoid this we remove the file
once we've finished processing it.
(From OE-Core rev: c08d7093bf654ffedb155c5627972e9ecfeb0b60)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently its hard to figure out if/when sstate is checking for possible
packages to speed up builds. This patch adds level 2 debug output which
better indicates what files are being searched for an why.
[YOCTO #1259]
(From OE-Core rev: a752b23767189f2678367e47ca0b41f49c56a631)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch removes the variables BASE_PACKAGE_ARCH, BASEPKG_HOST_SYS,
BASEPKG_TARGET_SYS and also removes the immediate assignments in
several core classes as these are no longer required.
This should make it clearer what some of the core variables do and
simplfy some overly complex and confusing class code.
(From OE-Core rev: d5521be2dcbaf213c140b0d12a4176380874426b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to check if we know of the task type, before we attempt
to process it. In order to reproduce the problem build with:
PACKAGE_CLASSES = "package_ipk"
Then change it to:
PACKAGE_CLASSES = "package_rpm"
Build again -- and then try bitbake -c cleansstate <recipe>
(From OE-Core rev: 8870ba9d261d3cacbe5d1219fdd95840c05ecf9e)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Distributions can then override this variable if needed
useful e.g. when building uclibc/eglibc both using
same tmpdir
(From OE-Core rev: 839e6e7f3717cc69cf8fc5b1c51bac63ec00655e)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix [YOCTO #964]
A recent commit 25a6e5f9(sstate: use only unique set of SSTATETASK) breaks
the ordered mapping between SSTATETASKS and SSTATETASKNAMES. As a result,
in sstate_cleanall, the line
taskname = tasks[namemap.index(name)]
gets an incorrect result, and "bitbake -c cleanall" doesn't really remove
the files populalted by do_populate_sysroot.
(From OE-Core rev: 2f6505f0e795b6c8cad641a6918739c3faac1f99)
Signed-off-by: Dexuan Cui <dexuan.cui@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* otherwise strange error like this:
ERROR: Logfile of failure stored in: /OE/shr-core/tmp/work/armv7a-oe-linux-gnueabi/libtool-cross-2.4-r1/temp/log.do_package_write_ipk.25551
Log data follows:
| ERROR: Package already staged (/OE/shr-core/tmp/sstate-control/manifest-nokia900-libtool-cross.deploy-ipk)?!
| ERROR: Function 'sstate_task_postfunc' failed
NOTE: package libtool-cross-2.4-r1: task do_package_write_ipk: Failed
ERROR: Task 11 (/OE/shr-core/openembedded-core/meta/recipes-devtools/libtool/libtool-cross_2.4.bb, do_package_write_ipk) failed with exit code '1'
is shown in this case with package_ipk twice in INHERIT
* Thanks to Richard for fix
(From OE-Core rev: f2fe5e840b8aa0558b5462ef2c7517b2f14ec2ea)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For safety, always delete the stamp files in sstate_clean regardless of
whether the manifest file exists or not.
(From OE-Core rev: f781c35da9a11eefdb06bda72ca89753df863efa)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Avoid deleting stamp files whose names contain the current task's name as a
substring. This will be especially important for example if do_package_write
is ever made an sstate task (as it would previously have deleted the stamps
here for do_package_write_ipk etc.)
(From OE-Core rev: ea743ea30e2289733d27979e8ec921648342da0e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When doing builds using sstate cache, there was no way to run
the equivalent of a pkg_postinst function. This is needed by
the SGML-related documentation recipes to properly update the
catalog files when new DTDs and stylesheets are installed.
SSTATEPOSTINSTFUNCS is a new variable you can set to function(s)
in your recipe to run after install is completed from sstate.
Thanks to Richard Purdie for suggesting this solution.
Signed-off-by: Scott Garman <scott.a.garman@intel.com>
Current sstate's fetch code doesn't reflect latest fetcher changes, so old
fetch style cause exception and fail silently.
[BUGID #708] got fixed.
Another issue is "import xxx" in python function from sstate.bbclass can only
sit in the head of the function, else have UnboundLocalError: "local variable
XXX referenced before assignment".
Signed-off-by: Zhai Edwin <edwin.zhai@intel.com>
This commit changes the sysroots path to be machine specific.
Changes includes:
1) STAGING_DIR_TARGET and STRAGING_DIR_HOST points to machine specific
paths.
2) task stamp files. Adding ${MACHINE} info into stamp files for
do_populate_sysroots and do_package tasks. Add a BB_STAMPTASK_BLACKLIST
to keep native, nativesdk, crosssdk, and cross-canadian stamp unchanged.
3) siteconfig path. Separate the site config path for different machines
to avoid one machine adopting the cache file of another machine.
4) sstate. Add machine name to sstate manifest file.
Change relocation code for sstate paths since sysroot is machine.
Keep native, nativesdk, crosssdk, and cross-canadian unchanged.
5) toolchain scripts. Change the environment path to point to machine
specific sysroots in toolchain scripts bbclass.
6) Relocate la files when populating to a different machine of the same
architecture.
7) Exclude STAGING_DIR_TARGET and STAGING_DIR_HOST parameter from sstate
siginfo since they contain ${MACHINE} information.
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Some steps (like gcc-cross-initial, gcc-cross-intermediate and eglibc-initial)
will install to new locations to avoid file overwriting.
Signed-off-by: Dexuan Cui <dexuan.cui@intel.com>
previously you could only build one set of packages for multiple machines:
MACHINE=foo bitbake task-base
MACHINE=bar bitbake task-base
would only create task-base packages for foo, but not for both foo and bar.
Doing MACHINE=bar bitbake task-base -c cleanall would remove the packages for foo.
The solution is to use MULTIMACH_ARCH as suggested by Richard Purdie.
Signed-off-by: Koen Kooi <koen@dominion.thruhere.net>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
This fixes the cleanall task to actually delete the sstage cache files. This
was not working correctly before due to attempting to use the current task's
hash in the package file name to be deleted, which of course is the hash of
the cleanall task when doing cleanall. The siginfo files are now deleted as
well.
Signed-off-by: Paul Eggleton <paul.eggleton@intel.com>
Remove the temp sstate-build-* directoies.
Take poky-image-minimal as an example, this saves about 17% disk space.
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
The fetcher has special handling for file:// mirror urls, being efficient and
just providing an updated path. Unfortunately the sstate fetching code wasn't
able to handle this. This patch detects this and injects a symlink to ensure
everything works. It also fixes some datastore references to be correct and
ensures the sstate download directory exists if it doesn't already.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
When installing a sstate package, directories tracked by plaindirs were being installed
to the incorrect location. With the current implementation this was limited to
the do_package task.
This patch ensures plaindirs tracked files are created in the correct location, fixing
the bug where these files would go missing.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
The directory list in the manifest file needs to be sorted such that child
directories are deleted first. Fortunately as the list is generated by walking
the directory tree achieving this is as simple as reversing the directory list.
Fixes [BUGID #269]
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Adds the ability to impose intercept function call(s) after
sstate_install and before sstate_package in sstate_task_postfunc.
Signed-off-by: Jeff Polk <jeff.polk@windriver.com>
dir name in manifest is incorrect:
/media/disk/builds/master-arm/tmp/sysroots/usr/
/media/disk/builds/master-arm/tmp/sysroots/lib/
/media/disk/builds/master-arm/tmp/sysroots/include/
Also add one debug note when removing manifest entries
Signed-off-by: Kevin Tian <kevin.tian@intel.com>
there's a bug in creating manifest_pattern, which means actually with
new sstate do_cleanup hasn't worked yet
We can use PSTAGE2_MANFILEPREFIX easily for this purpose.
Signed-off-by: Kevin Tian <kevin.tian@intel.com>
Using TARGET_ARCH as the pacakge architecture for sstate archives lead to some
issues when using two machines with different BASE_PACKAGE_ARCH's but the same
TARGET_ARCH such as netbook and qemux86 machines.
Probable fix for [BUGID #223]
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Joshua Lock
Mark Hatle
Richard Purdie
Martin Jansa
Khem Raj
Dexuan Cui
Paul Eggleton
Scott Garman
Zhai Edwin
Yu Ke
Dongxiao Xu
Koen Kooi
Paul Eggleton
Richard Purdie
Jeff Polk
Kevin Tian