Commit Graph

915 Commits

Author SHA1 Message Date
Michael Wood a6e4f94b01 bitbake: toaster: buildinfohelper Add the concept of CustomImagePackage
This adds the concept of CustomImagePackage this is similar to the way
layers and recipes work in that we have a set of data which is part of
the build history and a set of data which is part of the configuration
data that toaster uses to guide people in configuring their project. We
create a set of  built_packages for every build but only create a package
for configuration purposes if we don't already have one, so that the
CustomImagePackage only ever contains a unique list of packages that are
available to be added and removed from a CustomImageRecipe.

(Bitbake rev: f81bb65883baa6c0f8a4d48a4de3291a10543992)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:17 +00:00
Richard Purdie 602da7c921 bitbake: knotty: Don't show errors for universe provider issues
When running universe builds, we don't expect an error exit code for
provider warnings. Change the error messages to warnings in this case.

This deals with errors causing problems on our autobuilders amongst
other issues.

(Bitbake rev: d4989fb0355476de172169f0698757f7360e9a1f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-01 11:20:42 +00:00
Sujith H 3072361f86 bitbake: bitbake: BBUIHelper: Remove function findServerDetails
findServerDetails function can be removed safely
from the source tree. Couldn't find any files
calling this function.

(Bitbake rev: 46871f769db13ccd36deedc5b6f3dbc0a3d31c4b)

Signed-off-by: Sujith H <sujith.h@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-30 11:43:56 +00:00
Richard Purdie 5375e6431c bitbake: bitbake: Set process names to be meaninful
This means that when you view the process tree, the processes
have meaningful names, aiding debugging:

$ pstree -p 30021
bash(30021)───KnottyUI(115579)───Cooker(115590)─┬─PRServ(115592)───{PRServ Handler}(115593)
                                                ├─Worker(115630)───bash:sleep(115631)───run.do_sleep.11(115633)───sleep(115634)
                                                └─{ProcessEQueue}(115591)

$ pstree -p 30021
bash(30021)───KnottyUI(117319)───Cooker(117330)─┬─Cooker(117335)
                                                ├─PRServ(117332)───{PRServ Handler}(117333)
                                                ├─Parser-1:2(117336)
                                                └─{ProcessEQueue}(117331)

Applies to parse threads, PR Server, cooker, the workers and execution
threads, working within the 16 character limit as best we can.

Needed to tweak the bitbake-worker magic values to tell the
workers apart.

(Bitbake rev: 539726a3b2202249a3f148d99e08909cb61902a5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-30 11:43:56 +00:00
Elliot Smith 68f3e1e067 bitbake: toasterui: log OSErrorException metadata events
OSErrors occurring in toaster.bbclass are converted to
OSErrorException metadata events. They were then being swallowed
as unprocessed events by toasterui, which made them difficult
to spot.

Explicitly catch OSErrorException events and log them so they
are easier to spot and debug.

(Bitbake rev: 69f2b2bc373ce114609600b59a6b6ccef20771c9)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-18 22:06:46 +00:00
Elliot Smith fb94754e83 bitbake: toasterui: listen for bb.event.MetadataEvent
The event mask for toasterui doesn't include MetadataEvents.
This means that we're missing the ArtifactFileSize event
(among others), which is the one we use to populate the SDK
artifact table.

Add that event type to the toasterui event mask so we can
record SDK artifacts as they are created.

[YOCTO #7603]

(Bitbake rev: d0276a831bb8cffd42c8367895633eaa1fa1ed30)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-18 22:06:46 +00:00
Ed Bartosh 2ff4ccb13f bitbake: buildinfohelper: add provides info to the db
Added new entries to Provides model and link them to
Recipe_Dependency using 'via' field.

This data will be used by Toaster UI to show 'Provides:'
information for the recipes.

[YOCTO #6169]

(Bitbake rev: 336ddc8df611d4c8f1c3d3a06d0a85bb544c38bc)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:26:32 +00:00
Ed Bartosh 6a28ed33b7 bitbake: buildinfohelper: use providermap
Used providermap in store_dependency_information function
to find virtual dependencies. This should fix annoying
warnings "stpd: KeyError saving recipe dependency"

[YOCTO #6169]

(Bitbake rev: 85c416ca338c886db6e79651e44727482df9fb07)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:26:32 +00:00
Richard Purdie 46d62d076d bitbake: knotty: Use non-interactive mode as fallback for dumb terminals
TERM=dumb bitbake X

shows no output for task status which is suboptimal. Use the non-interactive
mode if the terminal doesn't support what we need for interactive mode giving
a better user experience. Also print a note to the console to say this has
happened.

[YOCTO #8768]

(Bitbake rev: 6f84cf4bd77f35fcd07e0b2f5149f1d6866a414d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-08 13:58:08 +00:00
Ed Bartosh 05b4fbc947 bitbake: uievent: refactor retry loop
Replaced 'while' loop with 'for' loop.
Made the code more compact and hopefully more understandable.

(Bitbake rev: 4e1e497c8432536b3522295e5b1284844ccea056)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:35 +00:00
Ed Bartosh ebc169c360 bitbake: uievent: get rid of EventHandler attribute
This attribute was introduced by mistake. EventHandle is used in the
code for the same purpose.

(Bitbake rev: 8d505ec8913a7d51de48b4f52bb64c5d6a0bb08e)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:35 +00:00
Ed Bartosh 4e0de6eca2 bitbake: uievent: add error to registerEventHandler return
Current code throws Exception("Could not register UI event handler")
if event handler can't be registered. The real reason of this is that
cooker is in busy state. Error message lacks information about this.

Added error message to the return value of registerEventHandler.
Included returned error message into the log message and exception
text.

(Bitbake rev: 07de1ca7d57dcd0cc37406feae2949da12a3fa7a)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:34 +00:00
Richard Purdie 522dcaa0cb bitbake: knotty: Improve exception error message
Instead of:
"""
can only concatenate tuple (not "int") to tuple
"""
we now see:

"""
Traceback (most recent call last):
  File "/media/build1/poky/bitbake/lib/bb/ui/knotty.py", line 324, in main
    termfilter.updateFooter()
  File "/media/build1/poky/bitbake/lib/bb/ui/knotty.py", line 210, in updateFooter
    lines = 1 + int(len(content) / (self.columns + 1))
TypeError: can only concatenate tuple (not "int") to tuple
"""

which makes tacking down and fixing the problem much easier.

Also ensure we set an error exit code.

(Bitbake rev: d965bcae6cfd268406a3bd1ef77c5bb6c6e1c6d7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 12:18:19 +00:00
Richard Purdie 01d67bfa77 bitbake: knotty: Fix row/column function return value issue
When the row handling was introduced, one of the callbacks was
missed resulting in:

TypeError: can only concatenate tuple (not "int") to tuple

Fix it.

(Bitbake rev: 0b77cea2bf5b5f5704e2650fb0332f5d78037781)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 12:18:19 +00:00
Richard Purdie 6c12efa5c2 bitbake: buildinfohelper: Update for buildstats layout change
This updates buildinfo helper for the recent buildstats layout change

(Bitbake rev: 30311bbe667e9f22de17fae00ff58da06a7c3e23)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 12:18:19 +00:00
Elliot Smith 23c48065d2 bitbake: toaster: Start Django machinery for database access
buildinfohelper, with the new import paths for our Django models
and Django 1.8, was not getting an active connection to the database.

In buildinfohelper, call django.setup() explicitly to make sure
that the database connection is ready and models can be queried
and saved.

[YOCTO #8364]

(Bitbake rev: 671aaab8cb7c494cd5c7621b45a6f41a203d8bb5)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:06 +00:00
Elliot Smith 7a0c45e478 bitbake: toaster: Create default project with get_or_create* method
Rather than maintain data as part of the migrations (as was
done for the default project previously), create the default
(cli builds) project on demand as a by-product of getting
it from the database.

[YOCTO #8364]

(Bitbake rev: 5fd8e90ab9b81d1bd0d301bc1c91228ecbbea74b)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:06 +00:00
Ed Bartosh 9de8dfa11a bitbake: toaster: Fix references to app paths
The buildinfohelper runs outside of Django, but still needs
access to our Django app classes.

Previously, the imports referenced the toaster.* app, which worked
fine. But in Django 1.8, this causes an error about the same
module being loaded multiple times from different paths.

Change the paths to our Django modules so they don't cause
this error to be thrown. We can do this as we've added our
application libraries to sys.path in the buildinfohelper anyway.

[YOCTO #8364]

(Bitbake rev: 070da64cf32c32b5ffc34d611b463c3a3960b419)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:06 +00:00
Ed Bartosh b322dec016 bitbake: toasterui: process SetBRBE event
Currently BRBE is known to toasterui only when build is
started. It's passed to it with BuildStarted event. This is
too late as if build fails earilier than build starts toasterui
can not inform Toaster about the failure.

Set BRBE as soon as it's provided by Toaster.
This should make toasterui to be able to inform Toaster
about early build failures, e.g. failures during recipe parsing.

(Bitbake rev: d7819508dac488a64be3caec88db285cda9599ab)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:06 +00:00
brian avery 7e8ff7b9d7 bitbake: toaster: toasterui Add ParseStarted/ParseProgress events to mask
Toaster is not able to see ParseStarted and ParseProgress events
for command-line builds. This means it's not possible for Toaster
to detect failed builds, if the failure occurs at a point before
the BuildStarted event, as the build won't show up at all.

Add these events to the event mask, so that Toaster's toasterui
can detect and respond to them.

(Bitbake rev: 16bfd3e3d145705a2b3a05648ddbcacc7a338dfa)

Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-09 09:03:35 +00:00
Richard Purdie f9384b0669 bitbake: knotty: Enforce terminal line limit to stop crazy scrolling
If there are more tasks running than there are lines on the terminal, the data
scrolls in ways the UI wasn't designed for. This patch adjusts the UI just to
show the processes which fit onto the number of rows in the terminal window.
You can see the total number running from the counter in the top left as usual
and this makes warning and errors messages scrolling from the top of the window
work as designed.

Ultimately, scrolling would be nice but is for another time, this fixes the
biggest UI issue on highly parallel machines.

(Bitbake rev: 67b77658e2bfa849f6f55c9c262cb11d6bfdb399)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-08 10:31:48 +00:00
Ed Bartosh 8b7a548e30 bitbake: toaster: get rid of complicated heuristics
Removed buildinfohelper code which was trying to guess layer version of
the recipe using build request information. The code caused creation of
duplicated recipes as it resulted in layer version from layer index
instead of returning build layer version. As a result of this Toaster
UI was not showing any information about recipes.

Default approach used to find layer version seems to work much better as
it finds proper layer version. Now toaster will use it as the only
way to find layer version.

(Bitbake rev: 101690bda7ad55dc0657483233c90c374713755b)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:11 +00:00
Michael Wood 53e981e583 bitbake: toaster: buildinfohelper Broaden the toaster created recipe data case
When build information is the 'original' source of the information we
need to return the recipe that was created rather than the copy of the
recipe that is taken for keeping build history. We do this already for
command line triggered builds, but we also have this case for custom
images. We can simply check if the built_recipe exists instead of
special casing this.

(Bitbake rev: 9a8653bf602b2111dee7ee6a459682a68a695b22)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:10 +00:00
Ed Bartosh 57e5f24e57 bitbake: toaster: do not create duplicate HelpText objects
buildinfohelper code expects only one HelpText object per
build/variable/description.
Current code creates more than one such an object, which causes
toastergui to crash with this exception:
 MultipleObjectsReturned: get() returned more than one HelpText -- it returned 2!

Used git_or_create API to ensure that only one HelpText object is
created.

(Bitbake rev: e9b46803eb6f1f4044919abf90c8aeb3536e73ed)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:10 +00:00
Ed Bartosh 990289511b bitbake: toaster: do not terminate bb server
Toaster needs bb server to be running all the time due
to merged analysis and managed modes. Server gets restarted
before every build triggered by UI, but it shouldn't be
terminated as it will influence command line builds.

[YOCTO #8279]

(Bitbake rev: e69c87c7842e796ffcd7193ecde22c8f688498f5)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:09 +00:00
Ed Bartosh 7c2b22556b bitbake: toaster: update brbe and project attributes
Updated attributes of buildinfohelper object as they can
be changed for every build. For example brbe is set by
runbuilds for every build triggered by Toaster UI.

(Bitbake rev: ea3bc8d01704dc64f6cb7b4f5fe66c312a575174)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:08 +00:00
Elliot Smith a0791c19db bitbake: toasterui: Create per-build logs
Rather than relying on bug 8411, which is conveniently creating
separate log files for each of our builds, create our own
log file for each build.

The log files are created in the same tmp directories that
bitbake users, but are timestamped to the millisecond to avoid
name collisions.

Each log file is opened on a ParseStarted event (for builds
triggered by Toaster) or BuildStarted event (for builds on the
command line: Toaster doesn't get the ParseStarted event
for command-line builds).

The log file is closed on the BuildCompleted event, or if the
build fails.

Because we start logging on ParseStarted for Toaster builds,
we're able to capture the "Build Configuration" section which
bitbake writes to output.

[YOCTO #8373]

(Bitbake rev: 7974203cd8bc66dff1fcc55f8723dedefaf72840)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-29 22:05:36 +00:00
Richard Purdie 5b19b71252 bitbake: bb/ui: Use getSetVariable command for BB_CONSOLELOG
Metadata can define BB_CONSOLELOG as containing ${DATETIME} and
this can get expanded to a different value each time the variable
is read. In the case of BB_CONSOLELOG, this behaviour is not
desireable.

The values of DATE/TIME are locked down at build time but this is too
late for the purposes of ensuring the system can figure out the real
value of BB_CONSOLELOG.

The best way to do this is to set the variable into the datastore, thereby
preserving its value.

[YOCTO #8411]

(Bitbake rev: 021f2eb55ab5863b57ed1b3f19f1b329bc1ad477)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-29 07:35:51 +00:00
Ed Bartosh 851f0d82d9 bitbake: toaster: add get_or_create_targets API
Target objects are created before the build if build is
started from UI in build mode. However, in analysis mode Target
objects don't exist and need to be created using information
from bitbake events.

Added new API call get_or_create_targets to retrive existing
target objects or create them if they don't exist yet.

(Bitbake rev: ef69be31d133696bde54605f5a18da660099734c)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-27 07:24:30 +00:00
Elliot Smith 93f0b61749 bitbake: toaster: Record critical errors
Critical errors (where a build failed for reasons of
misconfiguration, such as a machine being specified which is not
in a project's layers) were being ignored (only log records
up to ERROR level were being logged to Toaster's db). This meant that
the build would fail but would not correctly report why.

Add support for CRITICAL error levels to the LogMessage model,
include errors at this level in the errors property for a build,
and show errors at this level in the build dashboard.

[YOCTO #8320]

(Bitbake rev: b6eacbca9cacb607de864ab7d093deb296da8226)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:13:22 +01:00
Michael Wood ae82d778e6 bitbake: toaster: buildinfohelper Detect command line builds
When we're building using toaster as just a listener to bitbake
(analysis mode) we need to handle the case where the toaster configuration data
isn't present so we don't need to try and update the existing information.

(Bitbake rev: a22faae2c3a5948356ce3cbc73c34509de65d370)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:13:22 +01:00
Michael Wood 41286f4730 bitbake: toaster: buildinfohelper Skip packages we have no build info about
If there are more packages listed as installed than we know about from
bitbake, and therefore have insufficient information to be able to
create a Toaster Package object then skip it. Also handle the case where
a dependency references such a package.

Also clarify the error logging.

(Bitbake rev: b4ce793685f70cab3f28cb4329aaaf3878cd62e8)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 05:39:04 +01:00
Michael Wood 97d000677e bitbake: toaster: buildinfohelper associate build data with built_recipe
Make sure we associate build data with the built recipe rather than
toaster's configuration copy of the recipe.

(Bitbake rev: 34d4ef7289d72d151ad0acdccab8b99c8c31221e)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 05:39:03 +01:00
Ed Bartosh fa68ae0455 bitbake: toaster: use meaningful logging levels
Changed logging levels to more appropriate ones.

(Bitbake rev: 27d0360d13af0c698bf3a224b3f0d415f17bb678)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 05:33:01 +01:00
Ed Bartosh bd8b27b6a9 bitbake: toaster: ignore ReachableStamps event
Added ReachableStamps event to the list of known events to
ignore. This should stop toaster throwing error message:
ERROR: Unknown event: <bb.event.ReachableStamps>

(Bitbake rev: cd4137e13af6964858640b78aa7fe6f1612be251)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 05:33:01 +01:00
Michael Wood fe29297c6a bitbake: Revert "bitbake: toaster: don't re-create Target objects"
This delete followed up the foreign keys and deleted things that were
not expected to be deleted.

This reverts commit 08000eb27e.

(Bitbake rev: 46b119eb62a5a612fe4c0847862d34f408e556f7)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:38 +01:00
Michael Wood e6d967dba2 bitbake: toaster: buildinfohelper Create a copy of the built layer and recipe
Create a copy of the built layer and the recipes associated with it.
This is so that the user can view the historical information about a
build. i.e. a snapshot of the layer version and artifacts produced at
that build.

(Bitbake rev: 0683d9a2b15e3234a94437aaebac84bfcca1420b)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:38 +01:00
Michael Wood 922503f4c1 bitbake: toaster: Create a relationship between build information and toaster layers
Previously this layer relationship was done by trying to match path
information that came back to the buildinfohelper with trying to query for
data in toaster's layers table. This rarely matched due to the lose coupling.

(Bitbake rev: 838e77c7c3c4006abd1327343a004590ab652de9)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:38 +01:00
Elliot Smith 15b482b16b bitbake: toaster: Add fake entry to Target_File for filesystem root
The files-in-image.txt file is produced by bitbake after an
image is created, listing all the files in the image.
However, this list doesn't include the root directory ('/').

buildinfohelper.py then tries to construct the filesystem
tree from this file, assuming that every directory apart from
the root directory (which is special-cased) can be assigned
a parent. But because the root directory isn't listed in
files-in-image.txt, an object for the root directory is never
created.

The direct subdirectories of the root ('./bin', './usr' etc.)
then can't be assigned a parent directory, as the object
representing the root directory doesn't exist. This
results in a Target_File lookup error and causes the
directory listing page to fail.

Fix this by creating a fake entry for the root directory
in the Target_File table, so that the direct subdirectories
of / can be assigned a parent. Note that it doesn't matter
that the root is faked, as its properties are never shown
in the directory structure tree.

[YOCTO #8280]

(Bitbake rev: a4015768183e5a3fa39a6c2b4dea0088ca182d80)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-23 22:44:55 +01:00
Ed Bartosh 08000eb27e bitbake: toaster: don't re-create Target objects
Due to re-creating Target objects from bitbake events task information
stored in original objects is lost.

There is no valid reason to remove existing objects. It's safer to query
them instead of re-creating as original object contain more information
than events coming from bitbake.

(Bitbake rev: aab4aff75eefb31aa53885d7735feee5daa294aa)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-23 22:44:53 +01:00
Elliot Smith 8c63d6049d bitbake: toaster: Read correct cooker log path from toasterui
The BB_CONSOLELOG variable changes by the time we read it in
BuildInfoHelper. This means that the log file location we
are using is incorrect, so the links to the cooker logs don't
work.

Instead, read it at the point when the BuildStarted event occurs
in toasterui. The BB_CONSOLELOG variable has the correct value
here, so pass that to BuildInfoHelper.

[YOCTO #8209]

(Bitbake rev: 20609eebee0d2318806cf81913e7ce6dc1005507)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-18 09:05:33 +01:00
Elliot Smith 214ba674d0 bitbake: toaster: Improve how default project is identified and fetched
Command line builds are associated with a "default project"
(as we currently require a build to have a project). This
acts as a container for builds initiated outside Toaster.
Currently, this project is marked as the default by its ID
being 0. However, this doesn't work with MySQL, as MySQL
won't allow 0 in a foreign key which references an
autoincrement field.

Instead, use an is_default field to track the default Project
for builds initiated outside Toaster.

Add a method to fetch this default project, rather than fetching
a project with a magic ID.

Add this default project in a migration, rather than as a side
effect of a get_or_create() style method.

Also ensure that builds always have a project explicitly assigned
to avoid any magic with a build's project foreign key defaulting to
0 (as it no longer does).

[YOCTO #7932]

(Bitbake rev: 71b709a1bbc26d89d61873763b467d21e625b274)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-03 16:52:48 +01:00
Alexandru DAMIAN 25d4a35b03 bitbake: toaster logger: fix pylint issues
This patch fixes pylint issues in the toaster build data logger,
toasterui. The following types of warnings are touched here:

* fixing imports
* unused variables are set to _
* logger calls now use lazy evaluation instead of formatting
the string
* correct whitespace identation
* removes unneeded "pass" statements, and extra parantheses
* disable specific pylint warnings when decideing to override
them

(Bitbake rev: 947d47f15048baa967f88e03d80014e88ce152aa)

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-19 18:05:55 +01:00
Alexandru Damian c9455a7484 bitbake: toaster: fix pylint errors
Prompted by issues discovered during running pylint on
the toaster sources, this patch fixes:

* missing import in toaster ui
* improper call of function in toaster ui (logger.debug)
* improper function definitions in bbcontroller
* invalid references to objects in bldcontrol.models
* proper initialization of object fields in ToasterTables

Also inhibiting specific pylint errors in files where
the problems are mis-identified.

(Bitbake rev: 1c71955c416fb68455f7f70669aba4202c411807)

Signed-off-by: Alexandru Damian <alexandru.damian@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-19 18:05:55 +01:00
Paul Eggleton 715d857174 bitbake: Fix default function parameter assignment to a list
With python you should not assign a list as the default value of a
function parameter - because a list is mutable, the result will be that
the first time a value is passed it will actually modify the default.
Reference:

http://docs.python-guide.org/en/latest/writing/gotchas/#mutable-default-arguments

(Bitbake rev: 7859f7388f2e3f675d0e1527cfde18625f36f637)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-19 18:05:44 +01:00
Ed Bartosh a403ddbbba bitbake: bitbake: toaster: Wait for toaster gui to come
Set logging level for the Toaster logger to get messages
to toaster_ui.log

Reverted previous workaround.

[YOCTO: #7965]

(Bitbake rev: f1d8e0ff7233f17cff3a56db10d2443b392b407d)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-01 11:29:35 +01:00
Richard Purdie 4c2f28bf48 bitbake: cooker: properly fix bitbake.lock handling
If the PR server or indeed any other child process takes some time to
exit (which it sometimes does when saving its database), it can end up
holding bitbake.lock after the UI exits, which led to errors if you ran
bitbake commands successively - we saw this when running the PR server
oe-selftest tests in OE-Core. The recent attempt to fix this wasn't
quite right and ended up breaking memory resident bitbake. This time we
close the lock file when cooker shuts down (inside the UI process)
instead of unlocking it, and this is done in the cooker code rather than
the actual UI code so it doesn't matter which UI is in use. Additionally
we report that we're waiting for the lock to be released, using lsof or
fuser if available to list the processes with the lock open.

The 'magic' in the locking is due to all spawned subprocesses of bitbake
holding an open file descriptor to the bitbake.lock. It is automatically
unlocked when all those fds close the file (as all the processes terminate).
We close the UI copy of the lock explicitly, then close the server process
copy, any remaining open copy is therefore some proess exiting.

(The reproducer for the problem is to set PRSERV_HOST = "localhost:0"
and add a call to time.sleep(20) after self.server_close() in
lib/prserv/serv.py, then run "bitbake -p; bitbake -p" ).

Cleanup work done by Paul Eggleton <paul.eggleton@linux.intel.com>.

This reverts bitbake commit 69ecd15aece54753154950c55d7af42f85ad8606 and
e97a9f1528d77503b5c93e48e3de9933fbb9f3cd.

(Bitbake rev: a29780bd43f74b7326fe788dbd65177b86806fcf)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-01 07:34:09 +01:00
Leonardo Sandoval 3b5b74b430 bitbake: knotty.py: Make sure bitbake.lock is unlocked before exiting
Before exiting the UI, unlocks the bitbake.lock owned by cooker; this
way consecutive bitbake executions can lock it again without trouble.

[Yocto #7941]

(Bitbake rev: 69ecd15aece54753154950c55d7af42f85ad8606)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-07-09 18:00:21 +01:00
Michael Wood c86a0d290f bitbake: toaster: Fix build execution regression
Make sure the error value is initialised and update the string match
that we have to identify bitbake's current startup status.

Patch contribution from Eduard Bartosh

(Bitbake rev: 915ba08a8a3013e9787e564f2ffd8698c948f433)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-07-02 22:56:39 +01:00
Alexandru DAMIAN 583e320764 bitbake: toasterui: verify variable before usage
This patch verifies that BRBE is set before trying to use
it to read the checkout paths. This is needed for builds
ran outside Toaster control.

(Bitbake rev: e04807cd3135c9de96cc7f79245f329c24618b85)

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-06-26 09:27:32 +01:00