When using RPM, depends.dot may contain dependencies such as
"/bin/sh", which will confuse _toaster_load_pkgdatafile(). Ignore
them. While at it, also ignore dependencies that contain parentheses,
e.g., "libc.so.6(GLIBC_2.7)".
(From OE-Core rev: 80c117f46442ef442e34b7681ed3688789f505ac)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By using a single regular expression, the parsing of the depends.dot
file can be simplified a lot. This should also make it less
susceptible to formatting changes in that file.
(From OE-Core rev: 20684149bb659b34d3bcac8f202cb95d607567c1)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use the TaskArtifacts event to scan the SDK and ESDK manifests
to cleanly collect the respective artifact files.
The previous method was broken when the SDK file deployment moved
from the do_populate_sdk[_ext] tasks to their sstate tasks. That
method is disabled (but not yet removed) in preparation for the
rest of refactor work for the parent #10283 work.
[YOCTO #10850]
(From OE-Core rev: a6ec56d372f833be50f57bcb79b6ebe78bb93be0)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The recent change in buildhistory.bbclass to use a tab in
installed-package-sizes.txt between "KiB" and the package name caused
toaster_buildhistory_dump() to fail since it parses the file and
expected a space there.
(From OE-Core rev: 7b3692f57628b33840f6dbcddbe05ae56c24b98e)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.
Search made with the following regex: getVar ?\(( ?[^,()]*), True\)
(From OE-Core rev: 7c552996597faaee2fbee185b250c0ee30ea3b5f)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fire TaskArtifact MetaData event for deployment tasks when task
either completed or skipped. Event contains full task id
(recipe+task) and list of deployment artifacts from sstate
manifest.
This should allow Toaster to always get notified about deployment
artifacts produced by the build.
[YOCTO #9869]
(From OE-Core rev: 9b08503eabf78bc1b114416523b41dcce3449f58)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We can reach the method in toaster.bbclass which tries to read from
the files-in-image.txt file via a build which doesn't create that
file (e.g. "bitbake core-image-minimal -c rootfs"). This causes
the build to fail with an exception.
Check that this file exists before trying to read from it.
[YOCTO #9784]
(From OE-Core rev: 8b369cdd73ab17cdf834a591b97b25840caeb740)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
toaster.bbclass does a scan of the image deploy and SDK directories
when a build finishes. However, this brings no benefit and could be
better managed and made easier to modify if moved to toasterui and
carried out when the BuildCompleted event occurs.
Remove the image scan code from toaster.bbclass, prior to moving it
to toasterui and buildinfohelper.
Also remove the license manifest update code, as this can also be
done from toasterui.
The postfuncs for do_populate_sdk are retained, but no longer
do the directory scan for SDK artifacts. Instead, they fire
an event with the value of the TOOLCHAIN_OUTPUTNAME variable,
as this is only accessible at the point when the do_populate_sdk
and do_populate_sdk_ext tasks are run. The value of this can then
be used by buildinfohelper to find the SDK artifacts produced by a
target.
[YOCTO #9002]
(From OE-Core rev: 67ebb5406c0fcdd1b28bf446249aa6fe34a741a8)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In python3, strings are unicode by default. We need to encode/decode
from command pipelines and other places where we interface with the
real world using the correct locales. This patch updates various
call sites to use the correct encoding/decodings.
(From OE-Core rev: bb4685af1bffe17b3aa92a6d21398f38a44ea874)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The PACKAGES variable doesn't include all packages potentially
generated by a recipe, not least of all because it doesn't include
dynamic packages created via do_split_packages() or similar.
Instead of trying to guess which packages were generated, walk
${PKGDESTWORK}/runtime to find all package information that was
written to disk. This allows us to read all information for the
complete list of packages generated by the recipe.
For example before this patch we get SinglePackageInfo events for
9 packages from ncurses. With the patch applied we get 20 events
for all of the packages created during an ncurses build.
As a bonus we also switch to using the same postfuncs for both
do_packagedata and do_packagedata_setscene as they each result
in a PKGDESTWORK with the relevant directories and files
created.
[YOCTO #9115]
(From OE-Core rev: f1f012962a18d16b1cc22bde836cbb46fe9e6088)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Current code in toaster_buildhistory_dump assumes that bitbake
target doesn't contain task name. It uses target as a part of
path to the files with data that it analizes. It fails to find
files if target contains task name. Stripping task from the
target should solve this.
(From OE-Core rev: 901c4f96c87bb557e747245685b7942624915670)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were previously ignoring pkgs that came in from a setscene like from
an sstate mirror). With this patch we can use pkgs than come from sstate
for image customisation as well.
Also remove unused variable.
[YOCTO #9137]
(From OE-Core rev: 4ea7f96ae28ff53867f5ffa4158076761676d288)
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Clean up the code which gathers buildstats for Toaster,
and modify the field names so that the correct parts of the
buildstats files are used to derive the CPU usage values.
Also derive elapsed time for the build here, rather than in
Toaster, as we have ready access to the data in the correct
format.
[YOCTO #8842]
(From OE-Core rev: c145624ccc77ffd07a0b4c368c0dfce525e17eac)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
During refactoring of the SDK/artifact scan code in toaster.bbclass,
the code to find other non-image artifacts in the images/ directory
was incorrectly removed.
Reinstate that code and clean it up so it's clearer what's happening
and so that non-image artifacts are correctly reported.
[YOCTO #8956]
(From OE-Core rev: 4899041d59f3537c46eb79ba3471ca2b72caad89)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The postfunc for finding image files after completion of a build
fails, as the image files we're interested in don't exist at the
point when the scan is currently done (following do_rootfs).
Attach the postfunc for scanning for image files to the new
do_image_complete task, which definitely runs after the image files
have been created.
[YOCTO #8956]
(From OE-Core rev: 9c597ef05cec12178d886f83a4cf7070f032ab2f)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
SDK artifacts were being dumped alongside the image file dump.
However, the dump was not being triggered correctly, as it
should be attached as a postfunc to do_populate_sdk, rather than
do_rootfs, as do_rootfs doesn't always fire when populating
the SDK directory.
By adding a postfunc to do_populate_sdk, the postfunc is fired
in the correct situations.
Also fixes the directory which is scanned for SDK artifacts, as
this is different from the image file directory.
[YOCTO #7603]
(From OE-Core rev: cd2eb45ce1f9be15e133d6fe940c1cff2199099b)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Reading IO stats fails because the IO read/write bytes are
being converted to strings, then added to a numeric running
total.
Fix this by converting IO stats to integers.
(From OE-Core rev: 8e2475eecafc0161d25684f5b8239273739de759)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The buildstats changes removed the no longer needed get_bn() function,
replace this with references to BUILDNAME.
(From OE-Core rev: e1a37899da56014693f08d1c39cb6ec0a4ed2bf4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This updates the toaster class to use the new data format from buildstats.
This does mean it will no longer read IO data from older builds, however
since that data is completely useless anyway, I don't consider that to be
an issue.
(From OE-Core rev: 5c3987226cd9669d5fc0993d0541048475af9fcb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Running "bitbake xxx:do_unpack" resulted in exception at the
BuildCompleted event from toaster.
The reason for this exception was usage of ':' as a field
delimiter in toasterstatlist file. As target can optionally
contain ':<task>' suffix it caused split(':') to throw exception:
File "toaster_collect_task_stats(e)", line 71, in
toaster_collect_task_stats(e=<bb.event.BuildCompleted object at
0x7f8434deed50>)
ValueError: too many values to unpack
Fixed by changing delimiter ':' -> '::'
(From OE-Core rev: 03d01393d14b7b20dcb40ff89b1628883fd3b545)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch removes the symbolic links from the artifact list.
[YOCTO #7184]
(From OE-Core rev: cbb64f85dabefe267519e4b983d73c864b41e876)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Toaster will log all the files that it can find in the
deploy images directory, and list them, considering that
they may be artifacts of the current build not logged
anywhere else.
[YOCTO #6836]
(From OE-Core rev: 87a25b1dec86da809c9a13c10449bd33a471cb95)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixing the bug where the openembedded-core name was registered
as "meta" in toaster.
[YOCTO #6962]
(From OE-Core rev: ab9f17893c4b004906ec232da300915145c125e0)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We read the elapsed time fromt the build stats file, instead
of computing it independently.
[YOCTO #6833]
[YOCTO #6685]
(From OE-Core rev: 4f5a4ec0cdaf078463f04be4a6683816e9b78d5f)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch fixes an issue where the toaster postfuncs were
added to the task signature, making impossible the reuse
of sstate caches from builds outside toaster control.
Now the signatures do not differ between toaster and toaster-less
builds.
(From OE-Core rev: 552c5daceb9f51d2b7331a12dfa033f1ca3d7468)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Toaster may look up inexistent file paths in the build history
for packages that have been referenced but not built.
This triggers a failure, and this patch recovers by deleting
the reference to the non-built packages.
[YOCTO #6063]
(From OE-Core rev: 6bc4a0311cc3cd3f696d9dbca8fc0ef4e3bc340b)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Relaced the key with the join between path and file name.
[YOCTO #6090]
(From OE-Core rev: 85ffc93becb31772107a5a63b09fd3c16160f3ca)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add the PN to the buildstats event data in order to
proper select database task when multiple similar tasks
are executed for the same recipe file.
(From OE-Core rev: 43d717df16312a8f0333aff6b8b037b4b9d51bf3)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The license.manifest file is located in DEPLOY_DIR_IMAGE/licenses/
IMAGE_NAME dir. The data needed is collected after rootfs task.
[YOCTO #5649]
(From OE-Core rev: ff52c5ba15433f2b1e9723bf845e39da918ad59b)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We read the list of files in a built image and send it
over with the same event for packages in image.
(From OE-Core rev: 21bb659beca69c8bb379af2bf10afc843f529e57)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we can get the original package name when reading
package infos in toaster.bbclass, we save it for proper
referincing in the toater ui.
(From OE-Core rev: c3f3fed1b397b6ee6d44e7d39e63d084e4a88e30)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Toaster needs buildstats to be enabled in order to
collect task statistics. The toaster.bbclass didn't
have a guard to stop task data collection if the buildstats
weren't enabled, leading to a crash.
This patch verifies that the task variables are defined before
trying to use them.
(From OE-Core rev: 7f2637ec8d1b8840c31255572b5cb058777382e7)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the process of removing the local system accesses
from toaster UI (which must be able to run remotely),
the code to read package information is moved
from Bitbake Toaster UI to the server-side
toaster.bbclass
[YOCTO #5604]
(From OE-Core rev: 380d48da4b476f43554e38e464e7e25c930f88b6)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the process of removing the local system
accesses from toaster UI (which must be able to
run remotely), the code to read build stats
is moved from Bitbake Toaster UI
to the server-side toaster.bbclass
The code will accumulate a list of stat files
to be read at build completion. When the
build completes, the whole data list is read and
sent through in a single event.
[YOCTO #5604]
(From OE-Core rev: 0c455c0708335eecd1e659680b6cddb4782e80fa)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the process of removing the local system
accesses from toaster UI (which must be able to
run remotely), the code to read layer information
is moved from Bitbake Toaster UI
to the server-side toaster.bbclass
[YOCTO #5604]
(From OE-Core rev: 158679d244ff5b44354fb474c88122918b93a5b6)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The FILES_INFO entry in each pkgdata file stores the list of files for
each package. Make the following improvements to how this is stored:
* Store paths as they would be seen on the target rather than
erroneously including the full path to PKGDEST (which is specific to
the build host the package was built on)
* For simplicity when loading the data, store complete paths for each
entry instead of trying to break off the first part and use it as the
dict key
* Record sizes for each file (as needed by Toaster)
* Serialise the value explicitly using json rather than just passing it
through str().
Fixes [YOCTO #5443].
(From OE-Core rev: ca86603607a69a17cc5540d69de0e242b33382d3)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding a new bbclass that will collect and send relevant
data from the task context to the Toaster UI.
This bbclass consists of postfuncs that get executed
right after the main task func, and in the same context.
This allows data gathering in a synchronous manner during
the build, guaranteeing data integrity. This approach also
preserves the task signatures.
The data is moved to the UI through the event system.
There is no performance impact if the class is disabled.
License is MIT.
(From OE-Core rev: 1d2d37d579492b63d20ff8aa890a43b9a1576cf0)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Peter Kjellerstedt
David Reyna
Jochen Jaegers
Joshua Lock
Ed Bartosh
Elliot Smith
Richard Purdie
brian avery
Alexandru DAMIAN
Cristiana Voicu
Paul Eggleton