Writes build information to target filesystem on /etc/build such as enabled
layers, their current status and commit.
squashspaces was moved to oe/utils.py to make it available to different classes
and avoid code duplication.
[YOCTO #6770]
(From OE-Core rev: c9cc652e88ddedddf8a2f23fb9b8c024616526d7)
Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
SSTATEPOSTINSTFUNCS is now set with = in sstate.bbclass, and because
the line here in buildhistory.bbclass ends up being parsed before that
we now need to use _append or the value we are adding will be wiped out.
This fixes buildhistory no longer recording package information since
OE-Core revision 9d659c6f20fa4a141b491c62a3ef0dfb1f896d9c.
(From OE-Core rev: b5c05b474554925524073023f3aac2532c87537f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* use .txt extension like other reports (e.g. image-info.txt) it's
also useful when looking at the file from web server which doesn't
have text/plain as default content-type
* include whole BUILDCFG_HEADER/BUILDCFG_FUNCS instead of just layer
info
* this makes it easier for distro to include more fields without the
need to modify buildhistory.bbclass
* currently it adds following fields:
Build Configuration:
BB_VERSION = "1.23.1"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "Gentoo-2.2"
TARGET_SYS = "x86_64-oe-linux"
MACHINE = "qemux86-64"
DISTRO = "shr"
DISTRO_VERSION = "2013.07-next-20140725"
TUNE_FEATURES = "m64 core2"
TARGET_FPU = ""
(From OE-Core rev: 6db6c74f2876df7a183990426bb4262972966441)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The base_contains is kept as a compatibility method and we ought to
not use it in OE-Core so we can remove it from base metadata in
future.
(From OE-Core rev: d83b16dbf0862be387f84228710cb165c6d2b03b)
Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The old functions were calling the list_installed_packages() wrapper
function that only listed the packages in an image rootfs. Even for
target/host SDK. Also, a python crash was possible if 'bitbake -c
populate_sdk core-image-*' was called without calling 'bitbake
core-image-*' first. That's because the wrapper was always looking into
the image rootfs...
This commit fixes the problem and calls the right wrapper for image/sdk.
(From OE-Core rev: c1b1a6eb448aa1548e2ec669a9304b5a25bd8ba5)
Signed-off-by: Laurentiu Palcu <laurentiu.palcu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix task signatures for recipes that add to SSTATEPOSTINSTFUNCS changing
when adding and removing INHERIT += "buildhistory" (really this time!)
This relies on the BitBake vardepvalueexclude feature, however it will
not fail without it - signatures will be changed in that case though.
Part of the fix for [YOCTO #5897].
(From OE-Core rev: 27c8a9a282358b9a8a330252ee2104b250777b38)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bb.build.exec_func() does not allow passing arguments to the executed
scripts. Use wrappers instead.
(From OE-Core rev: c3dd44b751e351be52512e2aff885bf571c95fe5)
Signed-off-by: Laurentiu Palcu <laurentiu.palcu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Postfuncs are now incorporated into task signatures as of BitBake
rev b84d010144de687667cf855ddcb41c9b863c236e, so we need to exclude
the one we're adding to do_fetch. The change to the SSTATEPOSTINSTFUNCS
variable value was also entering the signatures of do_populate_sysroot
and probably other tasks, so we need to use a slightly hacky trick to
avoid that as well.
Apart from the final do_rootfs / do_populate_sysroot, this now means
that adding and removing INHERIT += "buildhistory" will not cause tasks
to be re-run.
Also update the copyright date, properly this time.
(From OE-Core rev: 953df67eb877a6d0fc68d122964440a9a47de3c3)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the package information contained unicode characters (for example, in
the package file listing) then writing them out as ASCII would fail, so
write them out using codecs.open() instead of open() using UTF-8. This
fixes ca-certificates failing in do_packagedata when buildhistory is
enabled.
(From OE-Core rev: dcf228fe69bfee4e22baad477ad407248c0f9cdb)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
OE-Core commit 259b8718a31b886f8a158aeb5de164840c9a28b2 fixed UTF-8
errors but broke decoding of escape sequences in strings (e.g.
pkg_postinst scripts had \n \t in them instead of newlines and tabs.)
We need a second call to decode() here as specifying 'string_escape' as
the second parameter won't do anything.
(From OE-Core rev: 15e0cdff08b8b9b826bbb9f00192a27318a3ee65)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The filenames sometimes may have strange names. With the 'awk' script
it handled a limited number of spaces in the filename and a package
installing a file named "test file with spaces" would have its name
truncated.
This patch uses the find's printf formating to simplify the code and
properly handle this case. From a testing image, the only diff produced
is:
,----[ files-in-image.txt diff ]
| --rwxr-xr-x root root 0 ./usr/bin/test\ file\ with\
| +-rwxr-xr-x root root 0 ./usr/bin/test file with spaces
`----
The options used are available since findutils 4.2.5, released in 19
Nov 2004, making it available in all supported host distributions.
(From OE-Core rev: b09e24449c1c9ae335732dd070eacf66777556a1)
Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
On systems where default locale is utf-8 we get errors like
File: 'buildhistory.bbclass', lineno: 38, function: write_pkghistory
0034: if pkginfo.rconflicts:
0035: f.write("RCONFLICTS = %s\n" % pkginfo.rconflicts)
0036: f.write("PKGSIZE = %d\n" % pkginfo.size)
0037: f.write("FILES = %s\n" % pkginfo.files)
*** 0038: f.write("FILELIST = %s\n" % pkginfo.filelist)
0039:
0040: for filevar in pkginfo.filevars:
0041: filevarpath = os.path.join(pkgpath, "latest.%s" % filevar)
0042: val = pkginfo.filevars[filevar]
Exception: UnicodeEncodeError: 'ascii' codec can't encode character
u'\xed' in position 337: ordinal not in range(128)
This patch specifies decode to use utf-8 so ascii and utf-8 based
locales both work
(From OE-Core rev: 259b8718a31b886f8a158aeb5de164840c9a28b2)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After 1b8e4abd2d9c0 [bitbake.conf/package: Collapse PKGDATA_DIR into
a single machine specific directory], oe-pkgdata-util does not use
target_suffix parameter, so do not need to loop the vendor
(From OE-Core rev: 03c5f39b4d7dd8c81e0a130b7d5884e5af039a24)
Signed-off-by: Roy Li <rongqing.li@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When writing out variable values to pkgdata, if the value has been set
in the datastore with an override for the package, we use the package
name override in the pkgdata key as well; however the recently added
code to read pkgdata in buildhistory.bbclass was just using the override
where we normally expect to have it. However, if a recipe overrides one
of the values that is normally set for the recipe on a per-package basis
(e.g. the external-sourcery-toolchain recipe sets PKGV this way) then
this led to KeyErrors. Re-write the pkgdata loading code to always strip
off the package name override if it is present.
(From OE-Core rev: e40e8e574b3688400a668d3ad76b6cef1920e3e0)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We don't normally perform any operations (such as "git pull") that
trigger "git gc --auto", thus garbage collection never happens which
means performance of accessing the repository degrades noticeably over
time. Add an explicit "git gc --auto" to clean things up when needed.
Thanks to Elijah Newren and Ross Burton for suggesting this.
(From OE-Core rev: 4a45a999e0ad2e99581428a5a6d34f483c00544f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Disable several python functions if not parsing within the worker
context. This avoids executing expensive operations while parsing
recipes (which is unnecessary).
(Thanks to Richard Purdie for pointing out the issue and suggesting the
workaround.)
(From OE-Core rev: 540a2a30be21c3eca4323efbe91e7dcfc31a4c97)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use a function added to SSTATEPOSTINSTFUNCS and read the necessary
information out of pkgdata, instead of using a function executed
during do_package that reads the data directly. This has two benefits:
* The package info collection will now work when the package content is
restored from shared state
* Adding/removing the inherit of buildhistory will no longer change the
do_package signatures and force re-execution of that function for
every recipe.
Fixes [YOCTO #5358]
(From OE-Core rev: cd7f7efcd5f297d876823b8f579ecefb9542b089)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add PACKAGE_EXCLUDE and NO_RECOMMENDATIONS to the info we track for
images, since these can change what ends up in the image.
(From OE-Core rev: a10189366f180b87f5be20b66834b7e7a9bb8c12)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This should have been removed when the implementation was rewritten in
OE-Core commit 2179db89436d719635f858c87d1e098696bead2a. The collected
values weren't being used anywhere since then.
(From OE-Core rev: cbc23a87c1897b7fda40f452dd36acb0bca3d197)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The buildhistory code has fallback paths for older bitbakes for now. The
distrodata class is much less used and it can be assumed a recent bitbake
is used in that case rather than adding fallback code.
(From OE-Core rev: 570cc145029fd9d5528aef5c27cb65164265c799)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we have a hierarchy of pkgdata directories and the code has to put together
a search path and look through each in turn until it finds the data it needs.
This has lead to a number of hardcoded paths and file globing which
is unpredictable and undesirable. Worse, certain tricks that should be
easy like a GL specific package architecture become problematic with the
curretn search paths.
With the modern sstate code, we can do better and construct a single pkgdata
directory for each machine in just the same way as we do for the sysroot. This
is already tried and well tested. With such a single directory, all the code that
iterated through multiple pkgdata directories and simply be removed and give
a significant simplification of the code. Even existing build directories adapt
to the change well since the package contents doesn't change, just the location
they're installed to and the stamp for them.
The only complication is the we need a different shlibs directory for each
multilib. These are only used by package.bbclass and the simple fix is to
add MLPREFIX to the shlib directory name. This means the multilib packages will
repackage and the sstate checksum will change but an existing build directory
will adapt to the changes safely.
It is close to release however I believe the benefits this patch give us
are worth consideration for inclusion and give us more options for dealing
with problems like the GL one. It also sets the ground work well for
shlibs improvements in 1.6.
(From OE-Core rev: 1b8e4abd2d9c0901d38d89d0f944fe1ffd019379)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were mangling names here to avoid characters that are invalid for dot
node names, but if you just quote all names that isn't necessary and we
retain the original naming, allowing easy searching of the graph files.
(From OE-Core rev: 99efdad886aa5063a68912846c00a46ba9b15536)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This makes it easier to obtain the difference from the last three
builds particularly where a single build might account for more than one
commit in the buildhistory git repository (e.g. if package and image
changes occurred).
(From OE-Core rev: 9512f3787295d662678c2943da31197c8df39e99)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Record the bitbake command line in the commit message as a further
context indication.
(From OE-Core rev: 34fb802b2170f0fe208e54e7a4d18f663db72f90)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* usually it's more important to know how much space will each
package take on target device then size of compressed package
* example for libewebkit0 with 4 different architectures, interesting
that om_gta02 .ipk is bigger but it's smaller when installed
before:
MACHINE DEFAULTTUNE SIZE (.ipk file)
om_gta04 cortexa8t-neon 15996 KiB libewebkit0
qemux86_64 x86-64 16992 KiB libewebkit0
spitz xscale 16148 KiB libewebkit0
om_gta02 arm920t 16260 KiB libewebkit0
after:
MACHINE DEFAULTTUNE SIZE (installed)
om_gta04 cortexa8t-neon 60544 KiB libewebkit0
qemux86_64 x86-64 63720 KiB libewebkit0
spitz xscale 60588 KiB libewebkit0
om_gta02 arm920t 56268 KiB libewebkit0
(From OE-Core rev: 85e4a77138381a6086d5ebd3a28cb5a94bc26a19)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since this variable is no longer used, we don't need to set a default
for it.
(From OE-Core rev: a5230835c539781b5b035dc6d0be3cac5a5bd305)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Multiple do_fetch[prefunc] can happen at the same time and
if not os.path.exists(pkghistdir):
os.makedirs(pkghistdir)
isn't safe
* Use bb.utils.mkdirhier which doesn't raise error when directory exists
(From OE-Core rev: 9aac194d7db79129dcbed29cfb89c57dccf33729)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* this way it's easier for distro to select if such issue should be
error, warning or ignored and also it gets recorded in qa.log
(From OE-Core rev: 44624853de8c88abdcae2ccd0902b390ff546d4c)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there's been a lot of changes the commit to buildhistory can take a
significant amount of time, so print a note so there's some indication of
progress.
(From OE-Core rev: ccd911ba8887f5c9a72d5178617c2a3ee73264ff)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
"xargs -n1 basename" will execute basename even if there are no lines
piped into it, causing a "basename: missing operand" error if no
packages are installed, which will happen for the target portion of
buildtools-tarball.
(xargs' -r option could have been used here, but it is a GNU extension
and I thought it best to avoid that for the sake of future
interoperability).
(From OE-Core rev: 03d86123e9e804e62de38effd307f070d7863080)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that bitbake supports masking events for event handlers, lets use
this so event handlers are only called for events they care about. This
lets us simplify the code indentation a bit at least as well as mildly
improving the event handling performance.
(From OE-Core rev: bff73743280f9eafebe4591f7368ead91a4eb74d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Data from SDKs produced from different images or different
meta-toolchain style recipes should go into separate directories,
since SDK_NAME doesn't include anything to differentiate these by
default (only pieces for the distro name and architectures).
Note you should delete or move existing data under the buildhistory/sdk/
directory manually after this change.
Fixes [YOCTO #4622].
(From OE-Core rev: 74658b66530184b0287152397341fbe411e90c71)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
"git add ." does not record files that were removed in the buildhistory
directory. Specify the -A flag to also record removals.
This was discovered by the following warning added in Git 1.8.3:
warning: You ran 'git add' with neither '-A (--all)' or '--ignore-removal',
whose behaviour will change in Git 2.0 with respect to paths you removed.
(From OE-Core rev: a45a247e2cfa58892a0c9eb050d603a38cd839db)
Signed-off-by: Jonathan Liu <net147@gmail.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adjust to the recent changes in the git fetcher. This code should work
with current and previous versions of bitbake and can be cleaned up when
we move to new bitbake versions.
(From OE-Core rev: 835b64d4e9ed7b627b56a75d529f1a403c5db1a8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a BUILDHISTORY_IMAGE_FILES variable specifying a space-separated
list of files within an image to copy into buildhistory, so that changes
to them can be tracked. Typically this would be used for configuration
files, and by default this includes /etc/passwd and /etc/group, but the
user is free to extend this list by setting the variable in local.conf.
Implements [YOCTO #4154].
(From OE-Core rev: ed6bb6e3db518082d3a9c45d548bc1339be2c5ca)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* like PERSISTENT_DIR I find BUILDHISTORY_DIR more persistent
then tmp-eglibc directory
* WARN: people with existing checkouts in TMPDIR/buildhistory
need to move it to new location first (in order to save history)
(From OE-Core rev: d3ad9a4c573fec08b5606a4c3e86f93cb95e8718)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Change do_write_srcrevs to a postfunc of do_fetch, avoiding a dependency
being created that causes large numbers of setscene tasks being executed
on every build with both buildhistory and rm_work being enabled.
(From OE-Core rev: a751e9042dfffcc5c4701634a1f1f598012d609c)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* persistent cache records tag-srcrev mappings, but is not shared between builders
* when tag is moved in remote repo, all builders should rebuild the component to
use the same source, show warning when revision is different than what was used
in last build
(From OE-Core rev: 0bc22ed6bd67031749e8f2cb5415dabf933eef56)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Collect SRCREV information in a separate task and write it out in a
format which is more consistent with the rest of the buildhistory
output. Using a task means that SRCREV values will also be recorded for
native recipes and not just target ones, and the new formatting also
correctly handles multiple entries in SRC_URI.
Also adds scripts/buildhistory-collect-srcrevs which will report on all
of the recorded SRCREV values in a format suitable for use in global
configuration (e.g. local.conf or a distro inc file) to override AUTOREV
values to a fixed set of revisions. Example output:
# emenlow-poky-linux
SRCREV_machine_pn-linux-yocto = "b5c37fe6e24eec194bb29d22fdd55d73bcc709bf"
SRCREV_emgd_pn-linux-yocto = "caea08c988e0f41103bbe18eafca20348f95da02"
SRCREV_meta_pn-linux-yocto = "c2ed0f16fdec628242a682897d5d86df4547cf24"
# core2-poky-linux
SRCREV_pn-kmod = "62081c0f68905b22f375156d4532fd37fa5c8d33"
SRCREV_pn-blktrace = "d6918c8832793b4205ed3bfede78c2f915c23385"
SRCREV_pn-opkg = "649"
Some notes on using this script:
* By default only values where the SRCREV was not hardcoded (usually
i.e. AUTOREV was used) are reported - use the -a option to see all
SRCREV values.
* The output statements may not have any effect in the face of overrides
applied elsewhere; use the -f option to add the forcevariable override
to each output line to work around this.
* The script does not do any special handling for multiple machines;
however it does place a comment before each set of values specifying
which triplet they belong to as shown above.
Relates to [YOCTO #3041].
(From OE-Core rev: 2179db89436d719635f858c87d1e098696bead2a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to get in after complementary package installation, so use
_append instead of +=.
(From OE-Core rev: 8be32b8f30f63691f6b7a9592361b0975c6f8d7a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These are mildly annoying when viewing git diffs of the buildhistory
repository, so let's just get rid of them.
(From OE-Core rev: c257f292bf75061647e380889487c7e2625592de)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
SDKs are constructed in a similar manner to images, and the contents can
be influenced by a number of different factors, thus tracking the
contents of produced SDKs when buildhistory is enabled can help detect
the same kinds of issues as with images.
This required adding POPULATE_SDK_POST_HOST_COMMAND and
SDK_POSTPROCESS_COMMAND variables so that data collection functions can
be injected at the appropriate points in the SDK construction process,
as well as moving the list_installed_packages and
rootfs_list_installed_depends functions from the rootfs_{rpm,ipk,deb} to
the package_{rpm,ipk,deb} classes so they can also be called during
do_populate_sdk as well as do_rootfs.
Implements [YOCTO #3964].
(From OE-Core rev: c3736064483d4840e38cb1b8c13d2dd3a26b36aa)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- create "latest_srcrev" for each recipe with the following format:
${BB_FILENAME},${SRC_URI},${SRCREV},${FROM_AUTOREV}
[YOCTO #3041]
(From OE-Core rev: f9acd756746e61b82b58d300f3ab13f3b086787d)
Signed-off-by: Constantin Musca <constantinx.musca@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the PR service is not enabled, the versions can go backwards and there
are cases we don't want to know about this. This patch adds a variable
allowing these checks to be turned off whilst still using buildhistory.
(From OE-Core rev: 49f33a42e93711c96fbb099f3e6ee7f092d7b356)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add RPROVIDES, RREPLACES, RCONFLICTS and RSUGGESTS to the list of
tracked variables. Of these, RPROVIDES is always output, whereas the
others are only output if they have a value (since it is more common
that they don't).
Implements [YOCTO #3391].
(From OE-Core rev: 564d76bed7b96d381d6438df81c0d5b4f5a7b2b0)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we do a signficant amount of tree traversal in many different places
which in inefficient. We can assume that the files don't change and cache the
file list which gives an efficiency improvement which this patch does using
a global variable.
(From OE-Core rev: 2d7608842d2dab07065e60aab729a5c8fd6b7907)
(From OE-Core rev: 5c7c8347eb1bc25d194be6f4be142ba0924e2600)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
oe-core removed the prerequisite to have sh as bash. POSIX doesn't define
any options and furthermore allows 'echo -e' to be the default behavior.
This means that in dash 'echo -e' will actually print '-e' and interpret
backslashes by default. We use instead 'printf' builtin command with or
without '\n' to simulate 'echo -e' or 'echo -n'.
'printf' needs format while 'echo' can be used without any arguments. So
'echo >' was replaced by 'printf "" >'.
'echo' without '-n' flag adds a new line by default so to keep the same
behavior of two new lines while using 'echo "\n"', 'printf "\n\n"' is
used.
[YOCTO #3138]
(From OE-Core rev: a19880ad10ccb5d7d909dcf9de5c3dc58a0ebcd3)
Signed-off-by: Andrei Gherzan <andrei@gherzan.ro>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Always write the metadata revisions for each layer into a machine-
readable "metadata-revs" file so that you can potentially link changes
in the output back to changes in the metadata. (Unlike the existing
similar build-id file, this is not specific to image changes.)
(From OE-Core rev: 45f38d167b523d769c072474b36b3d9b31b4e9d9)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Save PKG (the actual output package name, which is often different due
to debian renaming), and PKGE/PKGV/PKGR (which may be manipulated in
certain special cases e.g. gitpkgv.bbclass in meta-oe, the
external-sourcery-toolchain recipe, etc.) Note that these are only
written when they are different from the normal package name in the
case of PKG, or PE/PV/PR for the other variables.
Also, use PKGE/PKGV/PKGR instead of PE/PV/PR when comparing package
versions since these actually represent the version that the package
manager sees.
Implements [YOCTO #2787].
(From OE-Core rev: 65d7e9b2d4d8115ac9fd513c04f39a2df9556a5a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Write the value of these package script variables into the packageinfo
so that any changes to them can be tracked (in separate files since they
are multi-line).
Inspired by an earlier patch from Andreas Müller <schnitzeltony@googlemail.com>
(From OE-Core rev: 988a417c857c01c87de6ba9602968059cf8d830f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
getlastrecipeversion and readRecipeInfo weren't called anywhere.
Remove them so we don't have to keep them up-to-date with future
additions to the recipe info file.
(From OE-Core rev: 22bf6284576ccee607d7bd197420a29242df11c0)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a package is removed from PACKAGES, ensure that the package info file
and directory are removed from buildhistory so that we don't have stale
data lying around.
(From OE-Core rev: 223b183197b363edef0c11a25bad33555fca5a15)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Remove the ability to set BUILDHISTORY_KEEP_VERSIONS = "1" to save the
package history as flat versioned files rather than relying on git to
keep previous versions of the package information. git has proven to
work quite well in this capacity.
(From OE-Core rev: ccedfd250620fc562988ba730ad5717b107a9d3e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Reduce the number of calls to the packaging tool, especially in the case
of rpm, using helper utilities to gather the required information more
efficiently where possible.
(From OE-Core rev: d0b8a98c5b46c305afd389fc862b3bf0c6f1eaab)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bb and os modules are always imported so having these extra import calls
are a waste of space/execution time. They also set a bad example for people
copy and pasting code so clean them up.
(From OE-Core rev: 7d674820958be3a7051ea619effe1a6061d9cbe2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The echo line here was merging multiple lines into one, and the result
was that if both image and package changes had to be comitted then only
the image changes were being committed and the package changes could
potentially be merged into the next package change. Quoting the variable
reference fixes this.
Fixes [YOCTO #2411]
(From OE-Core rev: 2086bb86885951d0a74342e30ff205c24cb93f0d)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This seperates out image changes from package changes making the image diffs a lot easier to read.
(From OE-Core rev: fba198ac7efe476a25c5761878ef2fcee97bf9f1)
Signed-off-by: Koen Kooi <koen@dominion.thruhere.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When nothing has changed an empty commit prefixed with "No changes" will get generated so that the commit log of the buildhistory repo provides a complete log of all builds performed, not just those that resulted in changes.
(From OE-Core rev: 2e40558cc33c5c566a9a742c32eda3ea017f8607)
Signed-off-by: Koen Kooi <koen@dominion.thruhere.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are various conditions that lead to duplicate entries in the dot graph which need to get fixed, but this patch is a catchall. A previous attempt to address this only works on rpm which gives a \n seperated output, opkg doesn't.
Another benefit is that the sort order is now know, leading to less spurious diffs in buildhistory commits.
(From OE-Core rev: 479ae1e9b74aa2f04fb5da2f3541c3de0aa9de87)
Signed-off-by: Koen Kooi <koen@dominion.thruhere.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Avoid triggering a rerun of all do_package* tasks when OEBasicHash
signature generator is being used and the buildhistory package code or
any of the package related variables change.
Note that adding and removing INHERIT of buildhistory will still force
this rebuild - use the newly added BUILDHISTORY_FEATURES variable to
enable and disable buildhistory instead in order to avoid this.
(From OE-Core rev: 71b41cf7bc81a2767f379f16a408466130ad6523)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a BUILDHISTORY_FEATURES variable which can be set to "" to disable
buildhistory with the class still inherited.
BUILDHISTORY_FEATURES by default contains two items - image and package.
You can use these to disable the image and package history functions
individually - this is particularly useful if you want to get the image
contents and dependency graphs but don't need the package history.
Additionally, ensure we quit shell procedures gracefully by using return
instead of exit.
(From OE-Core rev: dd6a521045d5538a8ebf6775899d5e1319bea427)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ensure we do not erroneously split on + in RDEPENDS/RRECOMMENDS e.g.
libstdc++-dev was being split into libstdc and -dev.
(From OE-Core rev: cad533880df42ad4fe6f04d56d3a59cb4a033275)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For the purposes of querying the hostname to include it in the commit
message, it seems "cat /etc/hostname" does not work on the Yocto Project
autobuilder machines, and it's likely that the hostname command will be
more generally reliable, so use that instead.
(From OE-Core rev: cb939b753c9e7648a38e22e0349c279da785e69d)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The FILELIST order can vary depending on the order the files were
written which may change between builds with no ill effect, so sort the
list prior to writing it.
(From OE-Core rev: 2e9981000a211a89f88d7728393cc231e466581a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sort dependencies of each package which sometimes change order and cause
noise in the buildhistory repo, and at the same time remove duplicates
(which seem to be common especially for the RPM package query output).
(From OE-Core rev: 830df6067c1ea4a5aab580b42ba7e1e84fe1bcbf)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Values of image variables that are lists (e.g. IMAGE_INSTALL) are easier
to read if there are no extraneous spaces in them, so ensure that there
is only one space between each item.
(From OE-Core rev: 200159125eb6bcfc046c45cf5160b2eb340625e3)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sort DEPENDS, RDEPENDS, and RRECOMMENDS in package info files so that
any changes in order (which are not important) are smoothed out in the
change history.
Fixes [YOCTO #1961]
(From OE-Core rev: 06b740d4ca077fb4c89ee6d1065fabb02da45ec6)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sort the contents of files-in-image.txt to avoid unnecessary changes
showing up in the history due to reordering.
(From OE-Core rev: 791ad7a522f6800114febd1605072202c1450356)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Record some additional information about images - the uncompressed size
of the final image as well as the values of various variables that may
have influenced its contents. This is recorded in a machine-readable
"image-info.txt" file similar in structure to the package history files.
Also add some code to analyse changes to these values. (Most of the
variable values aren't monitored directly but will be used as contextual
information when they change at the same time as the content of the
image changing.)
(From OE-Core rev: 459ed6759a307b389f6ec1874136ec9aa0749120)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code that would have gone here has been superseded by the
buildhistory analysis functionality implemented in
meta/lib/oe/buildhistory_analysis.py and scripts/buildhistory-diff, so
remove it.
(From OE-Core rev: 5c8d583a12c507788c746637d5b4ddea62b0a629)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Just make it a bb.error when a package version goes backwards, it
doesn't make sense to fail the build immediately; the error(s) will
still be reflected in bitbake's exit code.
(From OE-Core rev: d3e70e0ed1ece5c8eb7f9c1c8140d3bd7b4333cf)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we're building on multiple hosts then it's useful to have the
hostname in the commit message.
(From OE-Core rev: abf3e7f7f56cc8bcdf104d8e27e7e366b0619ed6)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Check if BUILDHISTORY_DIR exists before doing anything with it, in
case no tasks that would have created it have executed
* Ensure the git repo in BUILDHISTORY_DIR is initialised with "git init"
before attempting to do anything with it
* Check if any files have been added or changed before adding and
committing, to avoid an error from "git commit"
(From OE-Core rev: 8eff4fea13317a741114f2a2e8e228d0155ba64c)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
--4ef4ab1d_66334873_12d0c
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
The layer list is multiline, and includes quotes, so including it within
quotes is problematic, particularly if the revision includes characters which
are not valid outside of a quoted string in shell. To reproduce this failure:
do a build with an scm layer not on a branch.
(From OE-Core rev: f74c500a0d3979dea7393524e71ca7ec03bb5fde)
Signed-off-by: Christopher Larson <chris_larson@mentor.com (mailto:chris_larson@mentor.com)>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Disable storing package history as version named files unless
BUILDHISTORY_KEEP_VERSIONS is set to 1; otherwise the adds of these
files that duplicate what is already in git anyway is just noise in the
git log.
(From OE-Core rev: fd2581770b8e4c42aa88f244daca58e27e11dff9)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Include package history collection from packagehistory.bbclass (thus
superseding it). The only change is to store package history under
BUILDHISTORY_DIR/packages and rename one of the functions.
(From OE-Core rev: c3266d138dc1cf18c0535bde5a9f48b1d3117bad)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create a new build output history reporting class, using testlab.bbclass
from meta-oe as a base. This records information from images produced by
the build process in text files structured suitably for tracking within
a git repository, thus enabling monitoring of changes over time.
Build history collection can be enabled simply by adding the following
to your local.conf:
INHERIT += "buildhistory"
The output after a build can then be found in BUILDHISTORY_DIR (defaults to
TMPDIR/buildhistory). If you set up this directory as a git repository and
set BUILDHISTORY_COMMIT to "1" in local.conf, the build history data will
be committed on every build.
(From OE-Core rev: 14acb530a27a3b088d0bfd56db291f4e72ace8ab)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit 508ff624fea705eb93cf2cc1e0c9c42cb817acf8. RP
accidentally pulled the wrong commit in, it was supposed to be a
*package*history bbclass change.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create a new build output history reporting class, using testlab.bbclass
from meta-oe and packagehistory.bbclass as a base. This records information
from packages and images output from the build process in text files
structured suitably for tracking within a git repository, thus enabling
monitoring of changes over time.
Build history collection can be enabled simply by adding the following
to your local.conf:
INHERIT += "buildhistory"
The output after a build can then be found in BUILDHISTORY_DIR (defaults to
TMPDIR/buildhistory). If you set up this directory as a git repository and
set BUILDHISTORY_COMMIT to "1" in local.conf, the build history data will
be committed on every build.
(From OE-Core rev: 508ff624fea705eb93cf2cc1e0c9c42cb817acf8)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alejandro Hernandez
Paul Eggleton
Martin Jansa
Matthieu Crapet
Otavio Salvador
Laurentiu Palcu
Ross Burton
Khem Raj
Roy Li
Richard Purdie
Jonathan Liu
Constantin Musca
Andreas Müller
Andrei Gherzan
Koen Kooi
Christopher Larson