Currently we do a signficant amount of tree traversal in many different places
which in inefficient. We can assume that the files don't change and cache the
file list which gives an efficiency improvement which this patch does using
a global variable.
(From OE-Core rev: 2d7608842d2dab07065e60aab729a5c8fd6b7907)
(From OE-Core rev: 5c7c8347eb1bc25d194be6f4be142ba0924e2600)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
oe-core removed the prerequisite to have sh as bash. POSIX doesn't define
any options and furthermore allows 'echo -e' to be the default behavior.
This means that in dash 'echo -e' will actually print '-e' and interpret
backslashes by default. We use instead 'printf' builtin command with or
without '\n' to simulate 'echo -e' or 'echo -n'.
'printf' needs format while 'echo' can be used without any arguments. So
'echo >' was replaced by 'printf "" >'.
'echo' without '-n' flag adds a new line by default so to keep the same
behavior of two new lines while using 'echo "\n"', 'printf "\n\n"' is
used.
[YOCTO #3138]
(From OE-Core rev: a19880ad10ccb5d7d909dcf9de5c3dc58a0ebcd3)
Signed-off-by: Andrei Gherzan <andrei@gherzan.ro>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Always write the metadata revisions for each layer into a machine-
readable "metadata-revs" file so that you can potentially link changes
in the output back to changes in the metadata. (Unlike the existing
similar build-id file, this is not specific to image changes.)
(From OE-Core rev: 45f38d167b523d769c072474b36b3d9b31b4e9d9)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Save PKG (the actual output package name, which is often different due
to debian renaming), and PKGE/PKGV/PKGR (which may be manipulated in
certain special cases e.g. gitpkgv.bbclass in meta-oe, the
external-sourcery-toolchain recipe, etc.) Note that these are only
written when they are different from the normal package name in the
case of PKG, or PE/PV/PR for the other variables.
Also, use PKGE/PKGV/PKGR instead of PE/PV/PR when comparing package
versions since these actually represent the version that the package
manager sees.
Implements [YOCTO #2787].
(From OE-Core rev: 65d7e9b2d4d8115ac9fd513c04f39a2df9556a5a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Write the value of these package script variables into the packageinfo
so that any changes to them can be tracked (in separate files since they
are multi-line).
Inspired by an earlier patch from Andreas Müller <schnitzeltony@googlemail.com>
(From OE-Core rev: 988a417c857c01c87de6ba9602968059cf8d830f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
getlastrecipeversion and readRecipeInfo weren't called anywhere.
Remove them so we don't have to keep them up-to-date with future
additions to the recipe info file.
(From OE-Core rev: 22bf6284576ccee607d7bd197420a29242df11c0)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a package is removed from PACKAGES, ensure that the package info file
and directory are removed from buildhistory so that we don't have stale
data lying around.
(From OE-Core rev: 223b183197b363edef0c11a25bad33555fca5a15)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Remove the ability to set BUILDHISTORY_KEEP_VERSIONS = "1" to save the
package history as flat versioned files rather than relying on git to
keep previous versions of the package information. git has proven to
work quite well in this capacity.
(From OE-Core rev: ccedfd250620fc562988ba730ad5717b107a9d3e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Reduce the number of calls to the packaging tool, especially in the case
of rpm, using helper utilities to gather the required information more
efficiently where possible.
(From OE-Core rev: d0b8a98c5b46c305afd389fc862b3bf0c6f1eaab)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bb and os modules are always imported so having these extra import calls
are a waste of space/execution time. They also set a bad example for people
copy and pasting code so clean them up.
(From OE-Core rev: 7d674820958be3a7051ea619effe1a6061d9cbe2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The echo line here was merging multiple lines into one, and the result
was that if both image and package changes had to be comitted then only
the image changes were being committed and the package changes could
potentially be merged into the next package change. Quoting the variable
reference fixes this.
Fixes [YOCTO #2411]
(From OE-Core rev: 2086bb86885951d0a74342e30ff205c24cb93f0d)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This seperates out image changes from package changes making the image diffs a lot easier to read.
(From OE-Core rev: fba198ac7efe476a25c5761878ef2fcee97bf9f1)
Signed-off-by: Koen Kooi <koen@dominion.thruhere.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When nothing has changed an empty commit prefixed with "No changes" will get generated so that the commit log of the buildhistory repo provides a complete log of all builds performed, not just those that resulted in changes.
(From OE-Core rev: 2e40558cc33c5c566a9a742c32eda3ea017f8607)
Signed-off-by: Koen Kooi <koen@dominion.thruhere.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are various conditions that lead to duplicate entries in the dot graph which need to get fixed, but this patch is a catchall. A previous attempt to address this only works on rpm which gives a \n seperated output, opkg doesn't.
Another benefit is that the sort order is now know, leading to less spurious diffs in buildhistory commits.
(From OE-Core rev: 479ae1e9b74aa2f04fb5da2f3541c3de0aa9de87)
Signed-off-by: Koen Kooi <koen@dominion.thruhere.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Avoid triggering a rerun of all do_package* tasks when OEBasicHash
signature generator is being used and the buildhistory package code or
any of the package related variables change.
Note that adding and removing INHERIT of buildhistory will still force
this rebuild - use the newly added BUILDHISTORY_FEATURES variable to
enable and disable buildhistory instead in order to avoid this.
(From OE-Core rev: 71b41cf7bc81a2767f379f16a408466130ad6523)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a BUILDHISTORY_FEATURES variable which can be set to "" to disable
buildhistory with the class still inherited.
BUILDHISTORY_FEATURES by default contains two items - image and package.
You can use these to disable the image and package history functions
individually - this is particularly useful if you want to get the image
contents and dependency graphs but don't need the package history.
Additionally, ensure we quit shell procedures gracefully by using return
instead of exit.
(From OE-Core rev: dd6a521045d5538a8ebf6775899d5e1319bea427)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ensure we do not erroneously split on + in RDEPENDS/RRECOMMENDS e.g.
libstdc++-dev was being split into libstdc and -dev.
(From OE-Core rev: cad533880df42ad4fe6f04d56d3a59cb4a033275)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For the purposes of querying the hostname to include it in the commit
message, it seems "cat /etc/hostname" does not work on the Yocto Project
autobuilder machines, and it's likely that the hostname command will be
more generally reliable, so use that instead.
(From OE-Core rev: cb939b753c9e7648a38e22e0349c279da785e69d)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The FILELIST order can vary depending on the order the files were
written which may change between builds with no ill effect, so sort the
list prior to writing it.
(From OE-Core rev: 2e9981000a211a89f88d7728393cc231e466581a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sort dependencies of each package which sometimes change order and cause
noise in the buildhistory repo, and at the same time remove duplicates
(which seem to be common especially for the RPM package query output).
(From OE-Core rev: 830df6067c1ea4a5aab580b42ba7e1e84fe1bcbf)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Values of image variables that are lists (e.g. IMAGE_INSTALL) are easier
to read if there are no extraneous spaces in them, so ensure that there
is only one space between each item.
(From OE-Core rev: 200159125eb6bcfc046c45cf5160b2eb340625e3)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sort DEPENDS, RDEPENDS, and RRECOMMENDS in package info files so that
any changes in order (which are not important) are smoothed out in the
change history.
Fixes [YOCTO #1961]
(From OE-Core rev: 06b740d4ca077fb4c89ee6d1065fabb02da45ec6)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sort the contents of files-in-image.txt to avoid unnecessary changes
showing up in the history due to reordering.
(From OE-Core rev: 791ad7a522f6800114febd1605072202c1450356)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Record some additional information about images - the uncompressed size
of the final image as well as the values of various variables that may
have influenced its contents. This is recorded in a machine-readable
"image-info.txt" file similar in structure to the package history files.
Also add some code to analyse changes to these values. (Most of the
variable values aren't monitored directly but will be used as contextual
information when they change at the same time as the content of the
image changing.)
(From OE-Core rev: 459ed6759a307b389f6ec1874136ec9aa0749120)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code that would have gone here has been superseded by the
buildhistory analysis functionality implemented in
meta/lib/oe/buildhistory_analysis.py and scripts/buildhistory-diff, so
remove it.
(From OE-Core rev: 5c8d583a12c507788c746637d5b4ddea62b0a629)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Just make it a bb.error when a package version goes backwards, it
doesn't make sense to fail the build immediately; the error(s) will
still be reflected in bitbake's exit code.
(From OE-Core rev: d3e70e0ed1ece5c8eb7f9c1c8140d3bd7b4333cf)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we're building on multiple hosts then it's useful to have the
hostname in the commit message.
(From OE-Core rev: abf3e7f7f56cc8bcdf104d8e27e7e366b0619ed6)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Check if BUILDHISTORY_DIR exists before doing anything with it, in
case no tasks that would have created it have executed
* Ensure the git repo in BUILDHISTORY_DIR is initialised with "git init"
before attempting to do anything with it
* Check if any files have been added or changed before adding and
committing, to avoid an error from "git commit"
(From OE-Core rev: 8eff4fea13317a741114f2a2e8e228d0155ba64c)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
--4ef4ab1d_66334873_12d0c
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
The layer list is multiline, and includes quotes, so including it within
quotes is problematic, particularly if the revision includes characters which
are not valid outside of a quoted string in shell. To reproduce this failure:
do a build with an scm layer not on a branch.
(From OE-Core rev: f74c500a0d3979dea7393524e71ca7ec03bb5fde)
Signed-off-by: Christopher Larson <chris_larson@mentor.com (mailto:chris_larson@mentor.com)>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Disable storing package history as version named files unless
BUILDHISTORY_KEEP_VERSIONS is set to 1; otherwise the adds of these
files that duplicate what is already in git anyway is just noise in the
git log.
(From OE-Core rev: fd2581770b8e4c42aa88f244daca58e27e11dff9)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Include package history collection from packagehistory.bbclass (thus
superseding it). The only change is to store package history under
BUILDHISTORY_DIR/packages and rename one of the functions.
(From OE-Core rev: c3266d138dc1cf18c0535bde5a9f48b1d3117bad)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create a new build output history reporting class, using testlab.bbclass
from meta-oe as a base. This records information from images produced by
the build process in text files structured suitably for tracking within
a git repository, thus enabling monitoring of changes over time.
Build history collection can be enabled simply by adding the following
to your local.conf:
INHERIT += "buildhistory"
The output after a build can then be found in BUILDHISTORY_DIR (defaults to
TMPDIR/buildhistory). If you set up this directory as a git repository and
set BUILDHISTORY_COMMIT to "1" in local.conf, the build history data will
be committed on every build.
(From OE-Core rev: 14acb530a27a3b088d0bfd56db291f4e72ace8ab)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit 508ff624fea705eb93cf2cc1e0c9c42cb817acf8. RP
accidentally pulled the wrong commit in, it was supposed to be a
*package*history bbclass change.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create a new build output history reporting class, using testlab.bbclass
from meta-oe and packagehistory.bbclass as a base. This records information
from packages and images output from the build process in text files
structured suitably for tracking within a git repository, thus enabling
monitoring of changes over time.
Build history collection can be enabled simply by adding the following
to your local.conf:
INHERIT += "buildhistory"
The output after a build can then be found in BUILDHISTORY_DIR (defaults to
TMPDIR/buildhistory). If you set up this directory as a git repository and
set BUILDHISTORY_COMMIT to "1" in local.conf, the build history data will
be committed on every build.
(From OE-Core rev: 508ff624fea705eb93cf2cc1e0c9c42cb817acf8)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie
Andreas Müller
Otavio Salvador
Andrei Gherzan
Paul Eggleton
Koen Kooi
Christopher Larson