Commit Graph

4766 Commits

Author SHA1 Message Date
Elliot Smith 4125da7763 bitbake: toaster: attach kernel artifacts to targets
The bzImage and modules files were previously attached to a build,
rather than to the target which produced them. This meant it was
not possible to determine which kernel artifact produced by a
build came from which target; which in turn made it difficult to
associate existing kernel artifact with targets when those
targets didn't produce artifacts (e.g. if the same machine + target
combination was built again and didn't produce a bzImage or modules
file because those files already existed).

By associating kernel artifacts with the target (via a new
TargetArtifactFile model), we make it possible to find all
the artifacts for a given machine + target combination. Then, in
cases where a build is completed but its targets don't produce
any artifacts, we can find a previous Target object with the same
machine + target and copy its artifacts to the targets for a
just-completed build.

Note that this doesn't cover SDK artifacts yet, which are still
retrieved in toaster.bbclass and show up as "Other artifacts",
lumped together for the whole build rather than by target.

[YOCTO #8556]

(Bitbake rev: 9b151416e428c2565a27d89116439f9a8d578e3d)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00
Elliot Smith e9808576da bitbake: toaster: improve image file suffix retrieval
Refactor retrieval of suffix from image file path, making it a
a method on Target_Image_File. This makes it easier to use this
in the build dashboard for individual images, plus reduces the
complexity of the code required to get all of the image file
suffixes for a build.

(Bitbake rev: 9c38de3dec74c122c2060cad37331bdafc6858ec)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00
Elliot Smith 5dfa120a7c bitbake: toaster: do image and artifact scan on BuildCompleted
Move the image and artifact scan code from toaster.bbclass and
consolidate its logic with the existing logic in buildinfohelper.

Remove handler setup for events which used to be fired from
toaster.bbclass but which are now handled directly by buildinfohelper.

[YOCTO #8556]

(Bitbake rev: f0085cd554604cfff4a3f40a34825fbb6878004f)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00
Elliot Smith b0585e6b0c bitbake: toaster: display Target targets in build dashboard
The build dashboard was showing the targets for the build in the page
heading and title as "Target object".

Add a filter which extracts the "target" from each Target object
as a string so that the heading and title display correctly.

Also sort the image file suffixes alphabetically.

[YOCTO #8556]

(Bitbake rev: 33a24992139e7dc05d14bbe1da60299b732945aa)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00
Ed Bartosh 844e1f9e1d bitbake: runqueue: improve exception logging
Runqueue errors direct the user to view the "failure below",
but no additional error message is available.

Log the stacktrace so that the user can see what went wrong.

Also fix a typo in the log message.

(Bitbake rev: e191f401e372ee181bc02250232ad9cb9a0e9477)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00
Robert Yang 7a220aeaae bitbake: lib/bb/utils.py: return sorted OrderedDict in explode_dep_versions2
The OrderedDict's item is sorted by insertion order, there might be a
problem when build the same recipe again, for example:
- First build of acl:
  Depends: libattr1 (>= 2.4.47), libc6 (>= 2.24)
- Second build of acl:
  Depends: libc6 (>= 2.24), libattr1 (>= 2.4.47)

They are exactly the same depends, but tools like "diff" doesn't think
so. Return sorted OrderedDict will fix the problem.

(Bitbake rev: a392f19f16ef8202ce3c12afbeb186a02438da17)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00
Paul Eggleton bc3a206de3 bitbake: knotty: avoid errors when fetching outside of a task
In a few places we use the fetcher code to fetch files outside of a
task, for example uninative in OE. In that case the pid of the event is
0 and that was causing an error in BBUIHelper.eventHandler(). Check the
pid and do nothing if it's 0.

(Bitbake rev: 59cb919e5cd5c653fb4d69b2d6a4320648443e10)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:50 +01:00
Ross Burton 2afcbfef2c bitbake: build: don't use $B as the default cwd for functions
When bitbake executes a shell or Python function it can cd/chdir() into a
directory before executing the task. If no directory is specified then the
default of $B is used.  However $B is an OpenEmbedded variable and BitBake
shouldn't be aware of it.

To solve this change the semantics slightly so that if no directory is
specified, the current working directory isn't changed.  There's also a sanity
check that emits a warning if a Python task does os.chdir() without restoring
the old path, and the previous working directory is restored.

This does change semantics: whereas before a function in OE would have $B as the
working directory unless specified, now the working directory is the top of the
build tree.  Any breakage this causes can be solved by either adding
do_some_task[dirs] = "${B}" or by using absolute paths in the task.

[ YOCTO #4634 ]

(Bitbake rev: 67a7b8b021badc17d8fdf447c250e79d291e75f7)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:50 +01:00
Ross Burton faa726824d bitbake: lib/bb/build.py: decode the command as UTF-8
The messaging FIFO is UTF-8, so decode the command as UTF-8 as well as the value
as otherwise "bberror" != b("bberror") and none of the messages from shell
functions are ever displayed.

Also add an else to the command parser so unhandled commands are noticed.

[ YOCTO #9947 ]

(Bitbake rev: 42d727743fa599e0a3c5ad2c29a1e6ede1a918bb)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:47:06 +01:00
Ross Burton b7a741cbca bitbake: lib/bb/utils: fix set_process_name
With Python 3 create_string_buffer needs a bytes() not a str() but as we were
catching all exceptions nobody noticed.

[ YOCTO #9910 ]

(Bitbake rev: 6576a9a95486c28a01d4211b4a33cc3e2c55a7cc)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:47:06 +01:00
Michael Wood 8dc159dd93 bitbake: toaster: tests Add selenium test for layerdetails page
This tests:
 - Adding remove layer from project
 - Deleting layer
 - Editing layer fields

(Bitbake rev: cabe7e2459fcd561bced2d39ba5bd173576153e5)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:29 +01:00
Michael Wood 8813726f1d bitbake: toaster: add Layer delete front end feature to layerdetails
Add the front end feature to delete a layer from the layer details page.

[YOCO #9184]

(Bitbake rev: 91815229f60eb9deba7d299f05c69b52ff1df59c)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:29 +01:00
Michael Wood e1ba2fd331 bitbake: toaster: api Add util function for returning the error response
Also clean up flake8 warnings in XhrBuildRequest

(Bitbake rev: aa0845242eda8650a97180bf6675551c26554cde)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:29 +01:00
Michael Wood 36dec688c7 bitbake: toaster: layerdetails api Fix saving of git revision of a layer
Update, clean up and move the api for updating a layerversion from the
views to api. Also update the layerdetails page to include the
layerversion id in the url getter.

[YOCTO #8952]

(Bitbake rev: 20f4e23bc86290f0a42881a7cac44c41eafa86fc)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:28 +01:00
Elliot Smith 903c3c2ef8 bitbake: buildinfohelper: ensure task datetimes are timezone-aware
When using toaster-eventreplay to run a bitbake event file
through toasterui/buildinfohelper, errors occur when the
tasks are updated with buildstats info:

  RuntimeWarning: DateTimeField Task.started received a naive
  datetime (2016-07-06 09:15:22.070000) while time zone support
  is active.

This is because a method in buildinfohelper returns a naive
datetime, but Django is expecting timezone-aware datetimes.

Ensure that datetimes used to set the started/ended times on
tasks are converted to timezone-aware datetimes.

(Bitbake rev: df9f4337bec87024ea6a43138c6080a755eb7fab)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:28 +01:00
Ed Bartosh 78b3fe6d5b bitbake: eventreplay: rewrite the script
Rewritten toaster-eventreplay to make code working as expected,
more compact and readable.

[YOCTO #9585]

(Bitbake rev: 45370a860b24a761d1b6e08ba752079cc45f54da)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:28 +01:00
Ed Bartosh 2b56c03264 bitbake: eventreplay: reorganize imports
Cleaned up module imports:
- Removed unused imports
- Removed import of print_function
- Removed duplicated imports
- Splitted importing bb.lib to 2 lines

(Bitbake rev: 332f5c9b20149e9f5757433df9458ce582ff32dd)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:28 +01:00
Ed Bartosh 9431a6f318 bitbake: eventreplay: replace MockConfigParameters with namedtuple
class MockConfigParameters has only one attribute and only __init__
method. Replacing it with namedtuple makes code less nested and more
readable.

[YOCTO #9585]

(Bitbake rev: 5d4df14b0d38f6c89ca16de6dada58b4bb015d71)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:28 +01:00
Ed Bartosh 70cc20daef bitbake: eventreplay: fix event loading code
Event objects are represented by base64-encoded strings in
the event file and can't be loaded by existing eventreplay code.

Fixed the code of loading events from file by decoding base64 strings
into the binary form and loading them with pickle.load.

[YOCTO #9585]

(Bitbake rev: a55c280c167f84caed6518119246e5a55f56cfd4)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:28 +01:00
Ed Bartosh 2199ff5ce6 bitbake: eventprelay: implement setEventMask command
Stored event mask list as self.eventmask for future use.
Fixed Exception: Command setEventMask not implemented.

[YOCTO #9585]

(Bitbake rev: 2e09074c70e89402de8f883dd402cd729118fc7e)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:28 +01:00
Ed Bartosh a1ceb0f9cd bitbake: eventreplay: add MockConnection.getEventHandle method
Fixed AttributeError: 'MockConnection' object has no attribute
'getEventHandle'

[YOCTO #9585]

(Bitbake rev: 6deae30480a4288da0c8b3529e61f3495f260f24)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:28 +01:00
Elliot Smith ed9c309e81 bitbake: toaster: fix layout for command-line builds in recent builds area
Command-line builds were displayed incorrectly, so that the
HTML elements for other builds were being "consumed" by the
command-line build elements due to incorrect positioning of
element end tags.

Fix by ensuring end tags close elements in the right places.

As the indentation was all over the place in the most recent
builds section template, it was almost impossible to see what the
problem was. So that was fixed, too.

[YOCTO #9842]

(Bitbake rev: 01659389813ad61d4f75b9f8d71528581322f0b0)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:28 +01:00
Ed Bartosh 7ed76e4c20 bitbake: bitbake-worker: don't reassign sys.stdout
Worker needs input stream in binary mode as it reads binary content
from it. Current code does it by detaching a buffer from sys.stdin
and assigning it back to sys.stdin.

Detached buffer is io.BufferedReader in binary mode. This operation
is implicit as its purpose is not easily understandable from the code.
Replacing it with fdopen(sys.stdin.fileno(), 'rb') should make the
code more understandable.

Assigning the buffer to sys.stdin is not needed as worker doesn't
use sys.stdin. Moreover, it leads to difficult to debug issues down
the stack. For example, devpyshell doesn't work without reopening
sys.stdin in text mode. This is not needed anymore after this fix as
sys.stdin is not changed in worker code and remains in text mode.

(Bitbake rev: b26bcff4c4d72775f1def7e769015464953b955c)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:28 +01:00
Michael Wood dab3b1b454 bitbake: toaster: views Fix most frequently built target in project reporting
Clean up and fix the most frequently built targets for the "Most built
recipes" section for the project configuration page.

[YOCTO #9846]

(Bitbake rev: 860475cfdd35301fb609ab3c89347566b0ca0adc)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:28 +01:00
Michael Wood 5018d5f095 bitbake: toaster: layerBtn avoid connecting handler to other build buttons
Some pages contain other build buttons which may have the same class
attached. Make sure that we only select the buttons in the tables where
layerBtn is used.

[YOCTO #9841]

(Bitbake rev: ec083fd943c6996307beb3be3421403870d2f2b7)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:27 +01:00
Sujith H 9d730f4747 bitbake: bitbake: toaster-tests: tests for project config
Add basic tests to validate the value, user types
in the text box for DL_DIR and SSTATE_DIR. Added
test case to validate the first char and inclusion
of space between the characters.

[YOCTO #9646]

(Bitbake rev: 1531e98c5ae1693d11d692c3589df29dff9364df)

Signed-off-by: Sujith H <sujith.h@gmail.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:27 +01:00
Sujith H dabb49be86 bitbake: toaster: fix validation checks for DL_DIR and SSTATE_DIR
Validation logic on the project configuration page
prevented a user from adding ${variable} to these paths.

Update validation so a user can see a better message
when they type characters into the text inputs.
Two types of validation are implemented.

Either:

* The value should start with a "/", to allow absolute paths.

or

* The value should start with a "$", to allow bitbake
   variables like ${TOPDIR}.

[YOCTO #9646]

(Bitbake rev: fca50153caef62a473d1812fd59ec7effe09f8c5)

Signed-off-by: Sujith H <sujith.h@gmail.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:27 +01:00
Sujith H be00eb29f3 bitbake: toaster: remove SSTATE_MIRRORS from projectconf
Remove SSTATE_MIRRORS from getting blacklisted. Hence the SSTATE_MIRRORS
should be removed from projectconf.html file so that users can know that
this variable is no more blacklisted.

[YOCTO #9598]

(Bitbake rev: b060eecae417521d8340748e7ace7a77488bd596)

Signed-off-by: Sujith H <sujith.h@gmail.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:27 +01:00
Sujith H f433664fa0 bitbake: toaster: remove SSTATE_MIRRORS from blacklists in views
Remove SSTATE_MIRRORS variable from blacklists in views.py
This helps user to point mirrors using SSTATE_MIRRORS with
toaster.

[#YOCTO 9598]

(Bitbake rev: c3dd4fb193c80ed9a6dbddf7f6a1eadd0964ac8c)

Signed-off-by: Sujith H <sujith.h@gmail.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:27 +01:00
Richard Purdie b301251668 bitbake: progress: Ensure missing start event is fired
The init function of the parent class fires a progress event for 0
progress rather than a start event. UI code was assuming that progress
events should always have a start event first. This change ensures that
the start event is correctly generated.

This fixes crashes that were seen in knotty in some configurations.

(Bitbake rev: 9841651e050a3e9f395ab3c62545c51197734584)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:27 +01:00
Richard Purdie 1b5688e0f0 bitbake: knotty: Handle process indicators more gracefully
Mistakes can happen with the generation of the progress events, change
knotty to be more tolerant of this rather than crashing, reporting to the
user when something unexpected happens. I haven't debugged why multiple
finish events appear to be triggered.

(Bitbake rev: 7dd06b1016b36420a9c55a45ff29dd64ae1dbcda)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:27 +01:00
Paul Eggleton 1b930b41a5 bitbake: runqueue: report progress for "Preparing RunQueue" step
When "Preparing RunQueue" shows up you can expect to wait up to 30
seconds while it works - which is a bit long to leave the user waiting
without any kind of output. Since the work being carried out during this
time is divided into stages such that it's practical to determine
internally how it's progressing, replace the message with a progress
bar.

Actually what happens during this time is two major steps rather than
just one - the runqueue preparation itself, followed by the
initialisation prior to running setscene tasks. I elected to have the
progress bar cover both as one (there doesn't appear to be much point in
doing otherwise from a user perspective). I did however describe it as
"initialising tasks".

(Bitbake rev: 591e9741e108487ff437e77cb439ef2dbca42e03)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:27 +01:00
Paul Eggleton f3b62c1c2e bitbake: runqueue: add ability to enforce that tasks are setscened
Add the ability to enter a mode where only a specified whitelist of
tasks can be executed outright; everything else must be successfully
provided in the form of a setscene task (or covered by a setscene task).
Any setscene failure outside of the whitelist will cause the build to
fail immediately instead of running the real task, and any real tasks
that would execute outside of the whitelist cause an immediate build
failure when it comes to executing the runqueue as well.

The mode is enabled by setting BB_SETSCENE_ENFORCE="1", and the
whitelist is specified through BB_SETSCENE_ENFORCE_WHITELIST, consisting
of pn:taskname pairs. A single % character can be substituted for the pn
value to match any target explicitly specified on the bitbake command
line. Wildcards * and ? can also be used as per standard unix file name
matching for both pn and taskname.

Part of the implementation of [YOCTO #9367].

(Bitbake rev: 624722c067a7fdd0c0f5d8be611e1f6666ecc4a2)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:27 +01:00
Paul Eggleton 76feac37ce bitbake: knotty: add quiet output mode
Quiet output mode disables printing most messages (below warnings) to
the console; however these messages still go to the console log file.
This is primarily for cases where bitbake is being launched
interactively from some other process, but where full console output is
not needed.

Because of the need to keep logging all normal events to the console
log, this functionality was implemented within the knotty UI rather
than in bb.msg (where verbose mode is implemented). We don't currently
have a means of registering command line options from the UI end, thus
the option actually has to be registered in main.py regardless of the
UI, however I didn't feel like it was worth setting up such a mechanism
just for this option.

(Bitbake rev: db95cdef08e339dec7462bfde3ad7d75c1c60dd8)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:27 +01:00
Paul Eggleton 465f93968f bitbake: knotty: show task progress bar
In addition to the "currently running n tasks (x of y)" message, show a
progress bar for another view on how much of the build is left. We have
to take care to reset it when moving from the scenequeue to the
runqueue, and explicitly don't include an ETA since not all tasks take
equal time and thus it isn't possible to estimate the time remaining
with the information available.

(Bitbake rev: de682015a3fefeff36ddc4197641a700f3fb558d)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:27 +01:00
Paul Eggleton 8d56d596bb bitbake: knotty: add code to support showing progress for sstate object querying
Add support code on the BitBake side to allow sstate.bbclass in
OpenEmbedded to report progress when it is checking for availability of
artifacts from shared state mirrors.

Part of the implementation for [YOCTO #5853].

(Bitbake rev: 070ae856da0715dbaf4c560c837ea796ffc29f00)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:26 +01:00
Paul Eggleton 0e3281f68b bitbake: lib/bb/progress: add MultiStageProgressReporter
Add a class to help report progress in a task that consists of multiple
stages, some of which may have internal progress (do_rootfs within
OpenEmbedded is one example). Each stage is weighted to try to give
a reasonable representation of progress over time.

Part of the implementation for [YOCTO #5383].

(Bitbake rev: 751b75602872a89e8b1a7c03269bc0fdaa149c6f)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:26 +01:00
Paul Eggleton ac5e720575 bitbake: lib: implement basic task progress support
For long-running tasks where we have some output from the task that
gives us some idea of the progress of the task (such as a percentage
complete), provide the means to scrape the output for that progress
information and show it to the user in the default knotty terminal
output in the form of a progress bar. This is implemented using a new
TaskProgress event as well as some code we can insert to do output
scanning/filtering.

Any task can fire TaskProgress events; however, if you have a shell task
whose output you wish to scan for progress information, you just need to
set the "progress" varflag on the task. This can be set to:
 * "percent" to just look for a number followed by a % sign
 * "percent:<regex>" to specify your own regex matching a percentage
   value (must have a single group which matches the percentage number)
 * "outof:<regex>" to look for the specified regex matching x out of y
   items completed (must have two groups - first group needs to be x,
   second y).
We can potentially extend this in future but this should be a good
start.

Part of the implementation for [YOCTO #5383].

(Bitbake rev: 0d275fc5b6531957a6189069b04074065bb718a0)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:26 +01:00
Paul Eggleton 1cf6e14a6c bitbake: knotty: import latest python-progressbar
Since we're going to make some minor extensions to it, it makes sense to
bring in the latest version of python-progressbar. Its structure has
changed a little but the API hasn't; however we do need to ensure our
overridden _needs_update() function's signature in BBProgress() matches
properly.

(Bitbake rev: c3e51d71b36cbc9e9ed1b35fb93d0978e24bc98a)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:26 +01:00
Paul Eggleton 481048cd2a bitbake: knotty: provide a symlink to the latest console log
If you're looking to find the latest console log repeatedly it can be a bit
tedious - let's just create a symlink just as we do with other logs to
make it easy to find.

(Bitbake rev: e9f41c0507a6527bf2ed86506813d4d4a89f8ebf)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:26 +01:00
Ross Burton 312f1a5e74 bitbake: fetch2/wget: attempt checkstatus again if it fails
Some services such as SourceForge seem to struggle to keep up under load, with
the result that over half of the autobuilder checkuri runs fail with
sourceforge.net "connection timed out".

Attempt to mitigate this by re-attempting once the network operation on failure.

(Bitbake rev: 54b1961551511948e0cbd2ac39f19b39b9cee568)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:26 +01:00
Robert Yang c6d50b2729 bitbake: lib/bb/build.py: remove task flag in deltask()
Otherwise the function like d.getVarFlag(e, 'task', True) which is used by
do_listtasks will still get it, and list the deleted tasks.

(Bitbake rev: 779d73619daf59f76f5b0313e7fb5409f6e82553)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-23 14:26:33 +01:00
Ed Bartosh 95f6e7bd0f bitbake: cooker: clean up EvertWriter
Restructured EventWriter code to make it more readable:
 - got rid of init_file method as it's called only once
 - renamed exception variable e -> err
 - renamed event variable e -> evt
 - simplified main 'if' structure of send method

(Bitbake rev: 31977e7bb98f676197c6cee66f6ab4c12d4dcbde)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-20 17:23:57 +01:00
Ed Bartosh f6fc25d1df bitbake: cooker: replace EventLogWriteHandler with namedtuple
class EventLogWriteHandler is a simple wrapper class with only one
class member. Replacing it with namedtuple makes code less nested and more
readable.

(Bitbake rev: 7c5b6812d32d173df36e7f9fc1d877329e79f994)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-20 17:23:57 +01:00
Ed Bartosh a158388a51 bitbake: cooker: don't remove event file
There is no need to remove output file as it gets rewritten by
open(self.eventfile, 'w') anyway.

(Bitbake rev: 1fc9957837b7038dfb983217a3fcd880f143e3a4)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-20 17:23:57 +01:00
Ed Bartosh e28b36e1e8 bitbake: cooker: encode event objects to base64
pickle converts python objects into the binary form that can't be
decoded to text and therefore can't be converted to JSON format.

Attempt to convert event objects raises this error:
TypeError:
    b'\x80\x03cbb.runqueue\nrunQueueTaskSkipped\nq\x00)...
    is not JSON serializable

Encoded pickled event objects to base64 to be able to convert data
structure to JSON.

[YOCTO #9803]

(Bitbake rev: f18055237e6084f90f6221442e3ba021dcc59c50)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-20 17:23:57 +01:00
Ed Bartosh 3e7edc303c bitbake: cooker: move EventLogWriteHandler to the top module level
EventLogWriteHandler object was created and used in
BBCooker.initConfigurationData.
This causes creation of multiple EventLogWriteHandler objects
and results in duplicated entries in the output event file
as BBCooker.initConfigurationData is called multiple times.

Added eventlogfile parameter to EventLogWriteHandler to avoid using
global variable DEFAULT_EVENTFILE.
Moved EventLogWriteHandler to the module level.
Created EventLogWriteHandler object in BBCooker.__init__ to ensure that only
one handler object is created.

(Bitbake rev: d3ad8eee850ec2df54aa09fae44cc7e69c12f32a)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-20 17:23:57 +01:00
Jussi Kukkonen 90d7b09af2 bitbake: depexp.py: port to gtk+3
The code is still a bit icky (and should be refactored to not use
Gdk.threads_enter/leave) but it should work about as reliably as
it did with Gtk+2.

Based on earlier patches by Maxin and Joshua.

(Bitbake rev: 8eee64a64144e27b5b8c2aca88e138882c3deab7)

Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-20 17:23:56 +01:00
bavery ce5523bf52 bitbake: toaster: remove last css file with bootstrap v2 license
Remove unused css file that still had the bootstrap v2 license in it.

(Bitbake rev: a8fe4177a6303aa57301c977c1daf7d4ff6ec586)

Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-16 23:54:19 +01:00
George McCollister f74ba25c0a bitbake: runqueue: Use tid instead of taskid in find_chains()
In 2c88afb6 find_chains()'s taskid argument was renamed to tid but
taskid is still used as key to explored_deps dictionary. Use tid instead
of taskid.

(Bitbake rev: 29a34ae8f5306d2779bcc761c52f1f9d13a0c0c5)

Signed-off-by: George McCollister <george.mccollister@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-16 22:42:53 +01:00