This patch fixes pylint issues in the toaster build data logger,
toasterui. The following types of warnings are touched here:
* fixing imports
* unused variables are set to _
* logger calls now use lazy evaluation instead of formatting
the string
* correct whitespace identation
* removes unneeded "pass" statements, and extra parantheses
* disable specific pylint warnings when decideing to override
them
(Bitbake rev: 947d47f15048baa967f88e03d80014e88ce152aa)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch verifies that BRBE is set before trying to use
it to read the checkout paths. This is needed for builds
ran outside Toaster control.
(Bitbake rev: e04807cd3135c9de96cc7f79245f329c24618b85)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We remove the "timespent", "errors_no" and "warnings_no" fields
in favor of computing the needed values at runtime. This prevents
inconsistencies in the UI.
Also removeing all references to BuildRequests from the interface -
all build details now display in the build dashboard.
Minor fixes related to data logging.
(Bitbake rev: 44f37394ed3e4ca02f940be172fe4395b0ee0f7d)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds logic to complete changing the interface
from showing BuildRequests to showing Build data.
The BuildRequest data is now transformed in Build data with
proper Toaster exceptions being recorded instead of listing
problems during startup as build errors.
(Bitbake rev: 51a41172d0b390370f9a38696b1ac65666ada4d2)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake logger now sets a default project if the TOASTER_BRBE
or TOASTER_PROJECT Bitbake variables are not set.
This a necessary step in getting all builds under a project,
as to unify the MANAGED and interactive modes.
Other small fixes are included, related to the size of the
fields in the database.
(Bitbake rev: 5e0bf388f4e5c1cc493ac8264785e631bad2f672)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This refactoring brings the "local_path" of the
layer from the Layer object to the Layer_Version object, which
is more appropriate as different checkouts of the same
Layer may live in different directories.
This enables us to store Recipe file paths relative to a Layer_Version
at all times, aleviating the need to store full file paths in the
database. We also turn the prefix of the path (e.g. virtual:native
path name space) into a pathflag field.
In turn, this solves the problem of mis-identification of tasks based
on the recipe file paths, since we can also match the namespace of the
file paths on the recipe files.
[YOCTO #7594]
(Bitbake rev: ec43dc569e370767c709dec225cbee0c99151c19)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch modifies the toasterui to properly return the exit
code based on the errors found in the toaster itself.
The upload event file API call will not delete event logs for which
toasterui showed an error. This will facilitate debugging.
Minor enhancement in the buildinfohelper to reduce the number
of lookups on unknown layer objects (prevented testing of the patch).
(Bitbake rev: 1ddd6a9e4280a4adf971132ff1fe7ec9b3252905)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch fixes the recipe and layer identification by path
when Toaster uses relative paths.
(Bitbake rev: a92bb33a3ceacab2bfee9df1c39a202832866970)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We improve logging and signalling of errors in the
toaster_ui.log to facilitate debugging on remote systems.
(Bitbake rev: 3cd248f99b90367bd41aab81e255fc1912434890)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We modify the toasterui to log relative recipe paths
in order to maintain consistency with data fetched from
the layer sources.
(Bitbake rev: 253d69e88fd68729196ad43c15e8733527d76198)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In build mode, instead of creating our own layer objects,
we identify the layer objects that the build system set up.
[YOCTO #7378]
(Bitbake rev: 22962b540ace6868cb357c0fd13f01ffd24449c4)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some databases, notably the SQLite3 adapter, force autocommits even if
the autocommit is turned off. The behavious is tracked in this bug:
http://bugs.python.org/issue8145#msg109965
Django refuses to work with autocommit off in this case, so we have
to take the same precautions when using manual transaction support.
[YOCTO #7363]
[YOCTO #7365]
(Bitbake rev: 90231ab63a129fa344d461c2911898ea0f07f206)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch fixes a problem where set sstate scene tasks
were not identified, causing cache attempt not being recorded.
[YOCTO #7223]
(Bitbake rev: 8a326a9a5a08981f1b7960e02fdb8a9436db16fb)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch disables autocommit for inserting build data,
effectively updating all build data in a single transaction.
This is a purely performance improvement patch, as the transaction
will always be commited.
Similar manual transaction handling in the layer source update
method. Added feedback messages during update method.
[YOCTO #7140]
(Bitbake rev: 3978c819e797f857235499a4b8ec238134f1c028)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch brings in cooker log saving and proper download links.
* toasterui will now write the cooker log file if running in managed
mode
* the BuildRequest has a new state, REQ_ARCHIVE, indicating that the
build is completed, and the artifacts are ready to be grabbed
* the runbuild test execution commands will gather needed artifacts,
and save them to a storage directory selected during Toaster setup.
* the build dashboard, project builds and all builds pages have
permanent links for the cooker log
[YOCTO #7220]
[YOCTO #7206]
(Bitbake rev: fad80e36c9da663b000cdf2cb3c75440c6431d84)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch improves the logging facilities for toaster in order
to help diagnose bugs that happen on user machines.
The logs are stored now under "/tmp/toaster_$$" where $$ is a
PID-based unique identifier. On shutdown, toaster will automatically
erase all logs unless errors are listed in the log file.
On error, Toaster provides suggestions on what to do.
This patch includes a minor fix found as a result of logging
improvements.
(Bitbake rev: 8a8248f7b7e30469f592e2f8adbf6ce21e8685c5)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
ToasterUI filters build artifacts by extension in order
to determine if a build artifact is an image or not.
Using IMAGE_FSTYPES for this purpose is not correct as
the varible value holding image extensions is just a coincidence.
So we just look if the filename contains the "rootfs" magic
string, which is a pretty good approximation.
[YOCTO #7213]
(Bitbake rev: b11e8bd626e0212ee72914529c3d92d1dd718674)
Signed-off-by: Alexandru Damian <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add a call that saves any queued events to the build
[YOCTO #7021]
(Bitbake rev: 4f5b19d453da64749affc1c27ec51b013bedc71a)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A bug slipped in the toaster ui that prevented saving of
build configuration despite the data being retrieved
from the server. This patch fixes the shaming mistake.
[YOCTO #7117]
(Bitbake rev: 8118f465b9f87c66b2a741008f69198ac5fea901)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds a "-w/--write-log" option to bitbake
that writes an event log file for the current build.
The name of the file is passed as a parameter to the "-w"
argument. If the parameter is the empty string '', the file
name is generated in the form bitbake_eventlog_DATE.json,
where DATE is the current date and time, with second precision.
The "-w" option can also be supplied as the BBEVENTLOG
environment variable.
We add a script, toater-eventreplay, that reads an event
log file and loads the data into a Toaster database, creating
a build entry.
We modify the toasterui to fix minor issues with reading
events from an event log file.
Performance impact is undetectable under no-task executed builds.
(Bitbake rev: 1befb4a783bb7b7b387d4b5ee08830d9516f1ac2)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We update and add logs throughout the code in order to help
with development. The extra logging is turned off by default,
but it can be enabled by using environment variables.
All logging happens through the Python logging facilities.
The toaster UI will save a log of all incoming events if the
TOASTER_EVENTLOG variable is set.
If TOASTER_SQLDEBUG is set all DB queries will be logged.
If TOASTER_DEVEL is set and the django-fresh module is available,
the module is enabled to allow auto-reload of pages when the
source is changed.
(Bitbake rev: 10c27450601b4d24bbb273bd0e053498807d1060)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add a BuildArtifacts class to store data about files
discovered during the build process and not stored anywhere
else.
Small cosmetic changes in the toasterui.
Add model methods to return file path display data relative
to the build environment instead of absolute file paths.
[YOCTO #6834]
(Bitbake rev: bbe24d912869312d561be199b2c029b0c898e049)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch brings in changes that allow a toasterUI coming in
from 'master' branch to record data from a 'daisy' or 'dizzy'
bitbake server.
This is needed to allow Toaster to record builds running
on older branch releases.
(Bitbake rev: 8d75e28e0688a6520311afce36543175f36910b3)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we have a managed build, we match the layers used for build
with the layers configured for project, as we know where the layers
are coming from
[YOCTO #6962]
(Bitbake rev: e02ec052a62cbc476bdac65cb7cea1167ce04781)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Changing ToasterUI to log toaster exceptions on a different level than
build errors.
Updating the build dashboard to show Toaster exceptions.
We add extra logging to console for exceptions.
Fixed a problem where packages database entries were created instead of
being looked up in the database, conficting with entries created to
satisfy dependency information.
Toaster now checks for invalid states at startup and performs needed
cleanups.
Removed loading reference to jquery-ui.min.css as we do not have this
file.
(Bitbake rev: 2378812bc24d433125fb940f110154f0ce638448)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Improve the performance of data logging in toasterui.
We modify the data queries used to:
* cache searching in memory
* insert in bulk (i.e. multiple values per insert, where possible)
On development test rig (networked mysql), on no-op build,
time for data recording is reduced from 4:10 to 1:30 (minutes).
We also improve the logging, so it is easier to detect
toasterui errors.
(Bitbake rev: d42784432f927f58730caf80546c66772e0fec89)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add improvements in data reading, following
issues discovered in testing.
- elapsed_time is now read from buildstats
- we add safeguards to not fail logging if the build was triggered
with a toaster_brbe configuration, but it's running in
1.6 mode
- added log markups for build finish to let other programs
known when the work is done.
[YOCTO #6833]
[YOCTO #6685]
[YOCTO #6887]
(Bitbake rev: 0b225035cefee3d3713a93f9a432e5e4d4e174f1)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We avoid a race between the setting the TOASTER_BRBE variable
and reading the variable in toaster ui by supplying the variable
at server startup time through the toaster.conf post-read file.
Additional small changes are included, including marking the
build request with the environment id of where the build took place.
(Bitbake rev: 7c333350418c4140e6c988c5272940f8057d327d)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We change the saving of the build id to the build request as
soon is the build is created, as to allow for a consistent
display of build data while build is in progress.
(Bitbake rev: 9504ca6a69ba6da21f88b3cc77fa5910d886b6c3)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to consistently use LogMessage.INFO/WARNING/ERROR to make sure toaster knows
how to categories these rather than passing in the "raw" loglevel value
which in best case comes from python logging but worst case any value.
[YOCTO 6885]
(Bitbake rev: 3aa13bc1d8218c97c76581a895fa5f03ff807fbe)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We refactor log saving to go through only one code path.
All logs that happened outside the build (i.e. before build
starting) now will be logged to either toaster_ui.log if the
build command ran in interactive mode, or to the build request
errors if the command ran in managed mode.
This enables proper display of error logs in project page.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch fixes a name collision on the event.data
in the MetadataEvent class. event.data is used in the
event system as a copy of "d" in certain situations,
and this collision triggered a bug leading to data loss.
[YOCTO #6332]
(Bitbake rev: 3f191b7cfe95aea4d4e96babf001d62d45dd3aaa)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patches fixes the build - project identification when
running under managed mode. The build is assigned to the
project from which it was triggered, and to the
build request, as to simplify relationships queries
in the database.
(Bitbake rev: af1d3373706d365f9138caec110fcb20a5966b7b)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add the build control functionality to toaster.
* The bldcontrol application gains bbcontroller classes
that know how to manage a localhost build environment.
* The toaster UI now detects it is running under build
environment controller, and update the build controller
database and will shut down the bitbake server once
the build is complete.
* The toaster script can now run in standalone mode,
launching the build controller and the web interface instead
of just monitoring the build, as in the interactive mode.
* A fixture with the default build controller entry for
localhost is provided.
[YOCTO #5490]
[YOCTO #5491]
[YOCTO #5492]
[YOCTO #5493]
[YOCTO #5494]
[YOCTO #5537]
(Bitbake rev: 10988bd77c8c7cefad3b88744bc5d8a7e3c1f4cf)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously, the buildinfohelper only set a django settings module
environment variable if none were set.
This may lead to problems when the starting the toasterui
from an already existing Django environment.
As such, we always override the variable to provide the
correct name for the local Django settings module.
(Bitbake rev: 8271e61a2fbddd3fc49556829675478d7505d58f)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some warnings and errors may come from a non-build context,
and they were not saved, even if they were counted for the current
build.
This patch saves these messages in memory until we have the
entire build context available.
[YOCTO #5642]
(Bitbake rev: fc7a74e7961775b5d7ff25298abed10138d24dc9)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We rename a local variable in as to prevent a conflict with
a similary named function parameter.
(Bitbake rev: f4e57f794651c4894600445e843ca9d5e104cd84)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you are using the testimage class then the old test mistook the
global inheritance of testimage.bbclass with the recipe inheriting
image.bbclass because it was only looking for that at the end of the
string.
Also tidy up the code so you an easily tell what it's doing. (The
original method may have been more "pythonic", but it does nothing for
readability.)
(Bitbake rev: b05e741cb5fe44b37538f2b727782f80dc9bb8fa)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These were not being collected properly because we were explicitly
excluding variables defined as functions from being stored in the
database. We don't want these to be shown in the variables list, and in
any case it makes sense for these to be stored elsewhere, so create a
separate model to store these.
Fixes [YOCTO #6050].
(Bitbake rev: 0d76a5461ce4bd554ff70a465064969e53edf0a4)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is an update on the sstate file saving data.
It saves both found and missed sstate tasks.
(Bitbake rev: 60c577b1080219b795d3c8ab4e149e929cf9ce14)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We save the missed sstate tasks as tasks that executed
but have the sstate_result set to "SSTATE_MISSED", signaling
that the attempt to find an sstate file failed.
(Bitbake rev: 6f22e02614adcc642fe011e5e31ca4936d1cb19d)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When saving task stats, if there were multiple tasks executed
based on the same recipe file, we might have saved the stats
to the wrong task by selecting another recipe.
This patch takes the PN into account to properly select
the file stats.
A check is also made to make sure we don't fail saving
data due to interrupted builds.
(Bitbake rev: e855031410daf2b99a6ca40b70956fe67c96f71c)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
File_name information of Target_image_file is being collected for a
.rootfs.manifest file. We would like not to collect this. The solution is
to cross check the information gathered for file_name with the content
of the IMAGE_FSTYPES variable. If any of the file_name entries does not
match the content of IMAGE_FSTYPES, we do not store it.
[YOCTO #5189]
(Bitbake rev: 017771ed0508b247edaf875789260906f44381f4)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch restricts the elapsed calculation to just
the events that have the "time" parameter set.
This fixes an error where data was lost due to an exception
where invalid dictionary lookups were made on the wrong
events.
(Bitbake rev: fa9f4eb8784553deb782bff34c5e04012c2c52c9)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The buildstats and toaster use separate time markers to measure the
duration of task execution. This causes a mismatch in the time
measured by buildstats class and the time measured in toaster.
The solution implemented here is to timestamp the creation of
every TaskBase event on the bitbake server side and calculate
the execution duration as the difference between creation time
of TaskSucceeded and TaskStarted events.
Based on an original patch by Marius Avram.
[YOCTO #5485]
(Bitbake rev: 7a08282c074c264f414cf7665dc873f51245072c)
Signed-off-by: Marius Avram <marius.avram@intel.com>
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch improves the recipe matching algorithm for
for matching recipes for native tasks.
(Bitbake rev: c350e4924abab8688c539608fd7f3af687d7265a)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patches fixes the inode type saved when writing the
target file list information.
(Bitbake rev: 9f34a1c5e94d73cdba1def7059c60211514e054c)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Based on image_name, the target is obtained, and the path
is added to the database.
[YOCTO #5649]
(Bitbake rev: 911b5191133956c30d53f57629c115db196b9ac8)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using ImageFileSize Metadata event, the image output file and its
size are populated into target_image_file table.
[YOCTO #5189]
[YOCTO #5228]
(Bitbake rev: a0b06d362b9aa08fda293489467af343c6ca6de4)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding code to write files-in-image data from the metadata
event to the database.
(Bitbake rev: f3a69ef7cc536a4b879d60936199a1002584c4f4)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds extra checks when selecting and writing
task and recipe objects to the database.
The patch fixes several issues where tasks may have been
misidentified between virtual-native and target tasks,
or spurious task objects may have been created.
(Bitbake rev: a6e597e690b3c6c6fa2af6db8cd871c02fc80421)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch fixes a logical error when updating task information
based on the corresponding sstate task state.
(Bitbake rev: 777458a20a7f686881e525a4d81b286c486ead6a)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch updates the database schema to resolve a
number of issues discovered while implementing the
UI interface.
We do not expect that all the data will come in valid
at this point.
[YOCTO #5453]
[YOCTO #5833]
[YOCTO #5836]
[YOCTO #5811]
[YOCTO #5812]
[YOCTO #5820]
(Bitbake rev: f8ad96d10a095e21fd2ce424c45e17f54642fb54)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add assert statements that validate expectations of
correct values and context when the data collection code is running.
These checks will help pinpointing unexpected data or
call flows, reducing debugging time.
Also contains a couple of very small fixes discovered through
these checks, including a virtual:native conflict with
regular tasks.
[YOCTO #5553]
(Bitbake rev: e2fbd5c6fa6b53514e2cb23d42aa639020d8a475)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We update the structure used to hold interm task data,
before it is written to the database, to lower the changes
of key collision.
This will also lead to a cleaner data structure and easier
inspection.
(Bitbake rev: 49cb9f543526a161bc4c097f94422ea08b491ef9)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This optimization is in support of the bug #5485. The function called
at the beginning of every build: store_dependency_information was taking
approximately 20sec and it was delaying the arrival of events from the
event queue. The change minimizes the calls to _save_a_task(),
reducing the time to half.
(Bitbake rev: b86fd2be40303d886fdb9ad3009355584d285acc)
Signed-off-by: Marius Avram <marius.avram@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to mark the package entries that are
created solely for dependency tracking purposes.
In order to avoid altering the database schema,
we mark the dependency targets with size = -1, since
this is not a valid size anyway and makes for easy
filtering.
[YOCTO #5803]
(Bitbake rev: d11ed273dd6c520b16e9ccfe79476f340006a55d)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to separate tasks with invalid states from the
no exec tasks, we add a new value OUTCOME_EMPTY for the tasks.
OUTCOME_EMPTY has the same value as OUTCOME_NA as to maintain
compatibility with already existing builds. New value for
OUTCOME_NA can be used to detect tasks with invalid states, i.e.
it should never appear after finishing a build.
Fixing noexec tasks outcomes.
[YOCTO #5763]
(Bitbake rev: 475643ad78796835bf2e731b9d0fa5794ec80dd1)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Under OE-Core, the name under which a package would
be installed in a target may have been different than the
name under it has been built or recorded in the dependencies
listings.
This patch addresses the way that Toaster records package
names, and adds the field of "installed_name" to save the
name under which a package have been installed in an image.
(Bitbake rev: 24e0367429b248108b104ab5a2af05efcf7a8c39)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
THis is a large set of fixes for the generic table, Build and
Dashboard pages.
Among the fixes:
* the table remembers which columns to show across refreshes,
based on saving the settings in a cookie
* added column timespent for a build which is a denormalization
of the completed_on - started_on information due to limits in
computing datetime differences in the SQL engine
* fixed formatting of the time differences
* various sorting header links fixed
* correct error and warning CSS classes applied to the
respective rows
* fixes multiple divide-by-zero error in displaying duration
estimations
(Bitbake rev: 61e3dee55ac577fce1c0ae0fe7e0d3cf644e8ae6)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch updates the task classification fields (outcome, etc)
as to
* Changes outcome names from SSTATE to CACHED and
from EXISTING to PREBUILT
* NoExec tasks now recorded as Not Executed / script type NA instead
of Executed / script type NOEXEC. Script type NOEXEC is deleted.
* SetScene tasks do not get order numbers
* New task method that returns a QuerySet for setscene tasks related
to this task: Task.get_related_setscene()
* New custom TaskManager that allows searching for setscene tasks
related to a certain task: Task.objects.related_setscene(task)
(Bitbake rev: a4164821a142f8b625a5fdc209adc6dc80874241)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
PE is an optional field in recipes specifiying the epoch
for the recipe. The canonical form for the
full recipe version string is: PE:PV-PR
If no PE is specified, we shouldn't store the initial ":"
character, as it leads to inconsistency with how the
version string is used elsewhere. This patch drops the leading ":"
[YOCTO #5459]
(Bitbake rev: f6031bd753917c459ab232d88d7dcfc3f10e8184)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Up until this patch, package information lived in two
places - one table for build packages and one table for
target installed packaged. This situation leads to
two problems: there is no direct link between a build
package and a installed package, and a lot of data is duplicated.
This change unifies all package types in a single table.
The SimpleUI remains the same for continuity sake,
but the REST API will be changed in a future patch.
The package dependencies and package files are now
kept in a single table.
Since we collect target installed package information at all times,
we need to expand it to supplement missing information if a
package is not actually built in the current build.
Small changes to the Simple UI reflect the updated database schema.
[YOCTO #5565]
[YOCTO #5269]
(Bitbake rev: f5d655bfaeb349c8680d74530617e34aa389d1f0)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the processes of removing local system accesses
from toaster UI, we remove the build stats
code that was moved to toaster.bbclass, and
adapt the database writing code to read the data
from BuildStatsList event sent by the toaster.bbclass
[YOCTO #5604]
(Bitbake rev: 4930ff5b471761c2a8d16c1935cdab9cf141d2d8)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the processes of removing local system accesses
from toaster UI, we remove the layer data reading
code that was moved to toaster.bbclass, and
adapt the database writing code to read the data
from event sent by the toaster.bbclass
[YOCTO #5604]
(Bitbake rev: 33b60a940f58e8374a8c7baa9bf28a98f54cdf13)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
That has no impact on the builds themselves, so we should just
remove that line from the variable history.
[YOCTO #5561]
(Bitbake rev: ae0ed55e80b7bd30c775b128b4114b306a50ff69)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
PKGSIZE is now in bytes in pkgdata, so we need to treat it as such in
the UI code for Hob / Toaster.
(Bitbake rev: 3b5ff814cd4a3efa4b17c6b343ec39c9acca5c9e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A typo in buildinfo helper leads to a bug where
information about tasks is not correctly stored.
This patch fixes the typo.
(Bitbake rev: 67b752993a2c64cba9ccc4fa662f0bddf081e74a)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to track the file where a configuration
variable was defined, this patch bring these changes:
* a new feature is defined in CookerFeatures, named
BASEDATASTORE_TRACKING. When a UI requests BASEDATASTORE_TRACKING,
the base variable definition are tracked when configuration
is parsed.
* getAllKeysWithFlags now includes variable history in the
data dump
* toaster_ui.py will record the operation, file path
and line number where the variable was changes
* toaster Simple UI will display the file path
and line number for Configuration page
There is a change in the models to accomodate the recording
of variable change history.
[YOCTO #5227]
(Bitbake rev: 78e58fed82f2a71f052485de0052d7b9cca53ffd)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
0 (not applicable) is not a valid sstate_result for tasks with
outcome 2 (sstate), which should return 3 (restored), 2
(failed) or 1 (missed).
Sstate_result for tasks with outcome 2 is equal to the outcome
of _setscene corespondent task.
[YOCTO #5220]
(Bitbake rev: 8ff8d75318ea88ba80c744b471e486901ef6749a)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The buildstats file path changes based on the
optional PE variable that may be defined for a
recipe.
The toasterui simply ignored the PE value, and
as such it didn't correctly reach buildstats files
for some of the tasks.
This patch fixes the issue.
[YOCTO #5073]
(Bitbake rev: 97b8ab88edc7c8dfb26b4cf305701ec96e52cc4f)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch addresses an issue where a failed sceneQueue task
entry was not updated on the Fail event. As a result, it
always showed the task as not-available.
[YOCTO #5216]
(Bitbake rev: 9b99a417f58381bac4bda412bcfd11de50403318)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The AUTHOR field in most recipes is not defined,
or it's not really consistently set in the metadata,
Also does it seem particularly useful.
This patch removes the AUTHOR variable from the
toaster system
[YOCTO #5449]
(Bitbake rev: da3ac049300be84defab7b32b0b99ab07c7d0a27)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Toaster needs to record information about packages
installed on a built target image, and dependencies
between these packages.
This patch fixes a bug where the variable from the
server wasn't read correctly leading which caused
the buildhistory to not be processed correctly.
Additionally, two display issues in the package table
were fixed, issues that lead to package information
being displayed incorrectly.
[YOCTO #5197]
(Bitbake rev: ab4bc18409d80de6d069e3dd76c3c54964fe5764)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the Configuration table, we need to link prefixed / suffixed
variables to the corresponding variable descriptions in documentation.conf.
[YOCTO #5198]
(Bitbake rev: 641d9c4fda5fe978154fdfab978c3c09e3906eab)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake tasks may be of type 'python' or 'shell',
or they may not be executed at all, which is record
as task type 'noexec'.
In order to record proper task type, this patch:
* creates no exec task type as the default value in
the toaster model definition
* adds full task flags to the bb.build.TaskStarted event
in build.py
* if the task actually starts, the toaster ui will
record the type of the task as either 'python' or 'shell'
based on the task flags.
[YOCTO #5073]
[YOCTO #5075]
[YOCTO #5327]
(Bitbake rev: 6648c57e6d369fc009ea3a9fe939def5d2c67bf5)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some of the data values may come of as None through the event system,
and the UI would encounter a problem saving the Configuration.
It would be trying to save these values as NULL in the
database, which is not allowed.
This patch adds more verification for data coming through
the event system.
Other minor updates:
* update for the event model from toaster.bbclass
* minor code flow fix in the event system
(Bitbake rev: 03fafd086381723c6486522873671515824e49f2)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding a new bitbake UI interface named 'toasterui'.
'toasterui' listens for events and data coming from a
bitbake server during a run, and records it
in a data store using the Toaster object model.
Adds a helper class named BuildInfoHelper that
reconstructs the state of the bitbake server and
saves relevant data to the data store.
Code portions contributed by Calin Dragomir <calindragomir@gmail.com>.
(Bitbake rev: 62200ff6694b21fbd5abf009a6f47ad93adf5309)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Alexandru DAMIAN
Michael Wood
Paul Eggleton
Cristiana Voicu
Marius Avram