In python3, strings are unicode by default. We need to encode/decode
from command pipelines and other places where we interface with the
real world using the correct locales. This patch updates various
call sites to use the correct encoding/decodings.
(From OE-Core rev: bb4685af1bffe17b3aa92a6d21398f38a44ea874)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The PACKAGES variable doesn't include all packages potentially
generated by a recipe, not least of all because it doesn't include
dynamic packages created via do_split_packages() or similar.
Instead of trying to guess which packages were generated, walk
${PKGDESTWORK}/runtime to find all package information that was
written to disk. This allows us to read all information for the
complete list of packages generated by the recipe.
For example before this patch we get SinglePackageInfo events for
9 packages from ncurses. With the patch applied we get 20 events
for all of the packages created during an ncurses build.
As a bonus we also switch to using the same postfuncs for both
do_packagedata and do_packagedata_setscene as they each result
in a PKGDESTWORK with the relevant directories and files
created.
[YOCTO #9115]
(From OE-Core rev: f1f012962a18d16b1cc22bde836cbb46fe9e6088)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Current code in toaster_buildhistory_dump assumes that bitbake
target doesn't contain task name. It uses target as a part of
path to the files with data that it analizes. It fails to find
files if target contains task name. Stripping task from the
target should solve this.
(From OE-Core rev: 901c4f96c87bb557e747245685b7942624915670)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were previously ignoring pkgs that came in from a setscene like from
an sstate mirror). With this patch we can use pkgs than come from sstate
for image customisation as well.
Also remove unused variable.
[YOCTO #9137]
(From OE-Core rev: 4ea7f96ae28ff53867f5ffa4158076761676d288)
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Clean up the code which gathers buildstats for Toaster,
and modify the field names so that the correct parts of the
buildstats files are used to derive the CPU usage values.
Also derive elapsed time for the build here, rather than in
Toaster, as we have ready access to the data in the correct
format.
[YOCTO #8842]
(From OE-Core rev: c145624ccc77ffd07a0b4c368c0dfce525e17eac)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
During refactoring of the SDK/artifact scan code in toaster.bbclass,
the code to find other non-image artifacts in the images/ directory
was incorrectly removed.
Reinstate that code and clean it up so it's clearer what's happening
and so that non-image artifacts are correctly reported.
[YOCTO #8956]
(From OE-Core rev: 4899041d59f3537c46eb79ba3471ca2b72caad89)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The postfunc for finding image files after completion of a build
fails, as the image files we're interested in don't exist at the
point when the scan is currently done (following do_rootfs).
Attach the postfunc for scanning for image files to the new
do_image_complete task, which definitely runs after the image files
have been created.
[YOCTO #8956]
(From OE-Core rev: 9c597ef05cec12178d886f83a4cf7070f032ab2f)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
SDK artifacts were being dumped alongside the image file dump.
However, the dump was not being triggered correctly, as it
should be attached as a postfunc to do_populate_sdk, rather than
do_rootfs, as do_rootfs doesn't always fire when populating
the SDK directory.
By adding a postfunc to do_populate_sdk, the postfunc is fired
in the correct situations.
Also fixes the directory which is scanned for SDK artifacts, as
this is different from the image file directory.
[YOCTO #7603]
(From OE-Core rev: cd2eb45ce1f9be15e133d6fe940c1cff2199099b)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Reading IO stats fails because the IO read/write bytes are
being converted to strings, then added to a numeric running
total.
Fix this by converting IO stats to integers.
(From OE-Core rev: 8e2475eecafc0161d25684f5b8239273739de759)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The buildstats changes removed the no longer needed get_bn() function,
replace this with references to BUILDNAME.
(From OE-Core rev: e1a37899da56014693f08d1c39cb6ec0a4ed2bf4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This updates the toaster class to use the new data format from buildstats.
This does mean it will no longer read IO data from older builds, however
since that data is completely useless anyway, I don't consider that to be
an issue.
(From OE-Core rev: 5c3987226cd9669d5fc0993d0541048475af9fcb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Running "bitbake xxx:do_unpack" resulted in exception at the
BuildCompleted event from toaster.
The reason for this exception was usage of ':' as a field
delimiter in toasterstatlist file. As target can optionally
contain ':<task>' suffix it caused split(':') to throw exception:
File "toaster_collect_task_stats(e)", line 71, in
toaster_collect_task_stats(e=<bb.event.BuildCompleted object at
0x7f8434deed50>)
ValueError: too many values to unpack
Fixed by changing delimiter ':' -> '::'
(From OE-Core rev: 03d01393d14b7b20dcb40ff89b1628883fd3b545)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch removes the symbolic links from the artifact list.
[YOCTO #7184]
(From OE-Core rev: cbb64f85dabefe267519e4b983d73c864b41e876)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Toaster will log all the files that it can find in the
deploy images directory, and list them, considering that
they may be artifacts of the current build not logged
anywhere else.
[YOCTO #6836]
(From OE-Core rev: 87a25b1dec86da809c9a13c10449bd33a471cb95)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixing the bug where the openembedded-core name was registered
as "meta" in toaster.
[YOCTO #6962]
(From OE-Core rev: ab9f17893c4b004906ec232da300915145c125e0)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We read the elapsed time fromt the build stats file, instead
of computing it independently.
[YOCTO #6833]
[YOCTO #6685]
(From OE-Core rev: 4f5a4ec0cdaf078463f04be4a6683816e9b78d5f)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch fixes an issue where the toaster postfuncs were
added to the task signature, making impossible the reuse
of sstate caches from builds outside toaster control.
Now the signatures do not differ between toaster and toaster-less
builds.
(From OE-Core rev: 552c5daceb9f51d2b7331a12dfa033f1ca3d7468)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Toaster may look up inexistent file paths in the build history
for packages that have been referenced but not built.
This triggers a failure, and this patch recovers by deleting
the reference to the non-built packages.
[YOCTO #6063]
(From OE-Core rev: 6bc4a0311cc3cd3f696d9dbca8fc0ef4e3bc340b)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Relaced the key with the join between path and file name.
[YOCTO #6090]
(From OE-Core rev: 85ffc93becb31772107a5a63b09fd3c16160f3ca)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add the PN to the buildstats event data in order to
proper select database task when multiple similar tasks
are executed for the same recipe file.
(From OE-Core rev: 43d717df16312a8f0333aff6b8b037b4b9d51bf3)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The license.manifest file is located in DEPLOY_DIR_IMAGE/licenses/
IMAGE_NAME dir. The data needed is collected after rootfs task.
[YOCTO #5649]
(From OE-Core rev: ff52c5ba15433f2b1e9723bf845e39da918ad59b)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We read the list of files in a built image and send it
over with the same event for packages in image.
(From OE-Core rev: 21bb659beca69c8bb379af2bf10afc843f529e57)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we can get the original package name when reading
package infos in toaster.bbclass, we save it for proper
referincing in the toater ui.
(From OE-Core rev: c3f3fed1b397b6ee6d44e7d39e63d084e4a88e30)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Toaster needs buildstats to be enabled in order to
collect task statistics. The toaster.bbclass didn't
have a guard to stop task data collection if the buildstats
weren't enabled, leading to a crash.
This patch verifies that the task variables are defined before
trying to use them.
(From OE-Core rev: 7f2637ec8d1b8840c31255572b5cb058777382e7)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the process of removing the local system accesses
from toaster UI (which must be able to run remotely),
the code to read package information is moved
from Bitbake Toaster UI to the server-side
toaster.bbclass
[YOCTO #5604]
(From OE-Core rev: 380d48da4b476f43554e38e464e7e25c930f88b6)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the process of removing the local system
accesses from toaster UI (which must be able to
run remotely), the code to read build stats
is moved from Bitbake Toaster UI
to the server-side toaster.bbclass
The code will accumulate a list of stat files
to be read at build completion. When the
build completes, the whole data list is read and
sent through in a single event.
[YOCTO #5604]
(From OE-Core rev: 0c455c0708335eecd1e659680b6cddb4782e80fa)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the process of removing the local system
accesses from toaster UI (which must be able to
run remotely), the code to read layer information
is moved from Bitbake Toaster UI
to the server-side toaster.bbclass
[YOCTO #5604]
(From OE-Core rev: 158679d244ff5b44354fb474c88122918b93a5b6)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The FILES_INFO entry in each pkgdata file stores the list of files for
each package. Make the following improvements to how this is stored:
* Store paths as they would be seen on the target rather than
erroneously including the full path to PKGDEST (which is specific to
the build host the package was built on)
* For simplicity when loading the data, store complete paths for each
entry instead of trying to break off the first part and use it as the
dict key
* Record sizes for each file (as needed by Toaster)
* Serialise the value explicitly using json rather than just passing it
through str().
Fixes [YOCTO #5443].
(From OE-Core rev: ca86603607a69a17cc5540d69de0e242b33382d3)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding a new bbclass that will collect and send relevant
data from the task context to the Toaster UI.
This bbclass consists of postfuncs that get executed
right after the main task func, and in the same context.
This allows data gathering in a synchronous manner during
the build, guaranteeing data integrity. This approach also
preserves the task signatures.
The data is moved to the UI through the event system.
There is no performance impact if the class is disabled.
License is MIT.
(From OE-Core rev: 1d2d37d579492b63d20ff8aa890a43b9a1576cf0)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie
Joshua Lock
Ed Bartosh
brian avery
Elliot Smith
Alexandru DAMIAN
Cristiana Voicu
Paul Eggleton