Commit Graph

4460 Commits

Author SHA1 Message Date
Ed Bartosh 6a28ed33b7 bitbake: buildinfohelper: use providermap
Used providermap in store_dependency_information function
to find virtual dependencies. This should fix annoying
warnings "stpd: KeyError saving recipe dependency"

[YOCTO #6169]

(Bitbake rev: 85c416ca338c886db6e79651e44727482df9fb07)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:26:32 +00:00
Ed Bartosh f2b7252190 bitbake: cooker: add providermap to dep_tree
Added providermap information to the result of buildDependTree API.

This will be used by Toaster to map virtual dependencies to recipes.

(Bitbake rev: d3e07368549f30265f59846a260efa8230a225ca)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:26:32 +00:00
Ed Bartosh 7e380d460c bitbake: taskdata: refactor get_providermap
Added optional parameter 'prefix' to filter out names that
don't start with specified prefix. Changed existing call
of get_providermap according to changed API.

Optimized the code: got rid of extra loop and temporary
list variable virts.

(Bitbake rev: df5a1392d6f91ccb44a99721c7d847da242121bb)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:26:32 +00:00
Richard Purdie 46731daa5f bitbake: main/runqueue: Add --setscene-only option to bitbake
Its turning out that we really need a way to have bitbake just run
the setscene tasks but not any real tasks, particularly for SDK
operations.

Add an option for this since its pretty straight forward. This allows
various nasty workarounds in OE-Core to be removed.

(Bitbake rev: e4a2aafa1650a227a04d92a8a0b31efaed2c310e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:26:32 +00:00
Richard Purdie 221d864042 bitbake: fetch/git: Change to use clearer ssh url syntax for broken servers
Some servers, e.g. bitbucket.org can't cope with ssh:// as part of
the git url syntax. git itself is happy enough with this but you
get server side errors when using it.

This changes the git fetcher to use the more common ssh url format
which also means we need a : before the path.

Seems a shame to have to do this due to broken servers however
it should be safe enough since this other form is the one most people
use on the commandline so it should be safe enough.

[YOCTO #8864]

(Bitbake rev: 4193e99adce8e88f12ac88d7578ad39575f7e346)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-08 13:58:08 +00:00
Richard Purdie 46d62d076d bitbake: knotty: Use non-interactive mode as fallback for dumb terminals
TERM=dumb bitbake X

shows no output for task status which is suboptimal. Use the non-interactive
mode if the terminal doesn't support what we need for interactive mode giving
a better user experience. Also print a note to the console to say this has
happened.

[YOCTO #8768]

(Bitbake rev: 6f84cf4bd77f35fcd07e0b2f5149f1d6866a414d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-08 13:58:08 +00:00
Ross Burton bfa7859ffa bitbake: cooker: fix findFilesMatchingInDir documentation
The documentation for findFilesMatchingInDir() was inconsistant with the
implementation: the regex was escaped before searching so effectively it's a
pure textual substring, and the machine example was broken.

(Bitbake rev: 6bef981488ec94b46dbe3797acfecf9c4b6ecbbc)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-08 13:58:08 +00:00
Ross Burton 3d4273716d bitbake: cooker: use in instead of count
No point counting all instances when we just want to know if there's any or not.

(Bitbake rev: 3369072efb653339da8dbd1ca864ff8e1ff899ca)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-08 13:58:08 +00:00
Richard Purdie f8185ff9f8 bitbake: ast: Add filename/lineno to mapped functions
Where we add in mappings for EXPORT_FUNCTIONS, add dummy filename
and lineno data so ensure the assumption that all python functions
have this is correct.

(Bitbake rev: 547128731e62b36d2271c4390b3fee2b16c535dc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:35 +00:00
Ed Bartosh a178c5a163 bitbake: main: kill server without queue setup
Setup of event queue includes registering of UI handler.
This operation can fail when cooker is busy. However, there is
no need in registering UI handler for terminating the server.

Moved the call of connection.terminateServer before setting up
of the event queue. This should make terminating server to work
more reliably as it doesn't depend on setting up the event queue
and registering UI handler anymore.

This should also help Toaster backend to restart bitbake server
and observer without getting "Could not register UI event handler"
errors.

[YOCTO #8776]

(Bitbake rev: 0c5a9349f797d05c282c2ada1893e187e05f0576)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:35 +00:00
Ed Bartosh 773700dbc3 bitbake: xmplrpc: split connect method
Current code in connect method sets up event queue, which requires
registering UI handler. This functionality may not be needed for
some operations, e.g. for server termination.

Moved functionality of setting up event queue in from 'connect'
method to 'setupEventQueue' in BitBakeXMLRPCServerConnection class.

(Bitbake rev: 4429871da76d6bd29e023ff42740fe7daa6b40fa)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:35 +00:00
Ed Bartosh 05b4fbc947 bitbake: uievent: refactor retry loop
Replaced 'while' loop with 'for' loop.
Made the code more compact and hopefully more understandable.

(Bitbake rev: 4e1e497c8432536b3522295e5b1284844ccea056)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:35 +00:00
Ed Bartosh ebc169c360 bitbake: uievent: get rid of EventHandler attribute
This attribute was introduced by mistake. EventHandle is used in the
code for the same purpose.

(Bitbake rev: 8d505ec8913a7d51de48b4f52bb64c5d6a0bb08e)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:35 +00:00
Ed Bartosh 4e0de6eca2 bitbake: uievent: add error to registerEventHandler return
Current code throws Exception("Could not register UI event handler")
if event handler can't be registered. The real reason of this is that
cooker is in busy state. Error message lacks information about this.

Added error message to the return value of registerEventHandler.
Included returned error message into the log message and exception
text.

(Bitbake rev: 07de1ca7d57dcd0cc37406feae2949da12a3fa7a)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:34 +00:00
Ed Bartosh 01419d5565 bitbake: cooker: add state.get_name method
Added method to convert state code into the human readable name.
It will be used in logging and error reporting.

(Bitbake rev: 9ec6379b27d210214d0b3f2e55962f721b7f5f51)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:34 +00:00
Juro Bystricky 763506d49b bitbake: fetch2/__init__.py: Add support for 7-Zip
7-Zip is a file archiver claiming the highest compression ratio.
This patch allows using 7-Zip commpressed files in bitbake recipes.
Two common formats are supported:

SRC_URI = "file://abc.tar.7z"
SRC_URI = "file://abc.7z"

(Bitbake rev: 7120f5bfaae54e91bc95da5667831424724ce613)

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:34 +00:00
Richard Purdie f5bfc1cc26 bitbake: utils: Remove double compile from better_compile
Poking around the ast to correct linenumbers works well for runtime failures
but not for parsing ones. We can use blank linefeeds to correct the line
numbers instead, with the advantage that we don't need to double compile.

(Bitbake rev: 10256ac3e7be7e691176ecc5d55856d88f1fe940)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:34 +00:00
Robert Yang b4141f6494 bitbake: fetch2/local.py: avoid using PREMIRROR
The PREMIRROR isn't useful for "file://", so avoid using it, this is
good for searching speed and can reduce useless lines in log.do_fetch.

(Bitbake rev: e130dca85bac82bd4d88f94a6bf9fe36e8ad4d7c)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:34 +00:00
Richard Purdie 1ad3595e8c bitbake: siggen: Change exception note into a warning
This makes no sense as just a note, its at least a warning and useful
to get an idea of which codepath is failing.

(Bitbake rev: 0194cf0da24dc72dab0612cd54aa5190e6cd92f2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:34 +00:00
Richard Purdie 4ba49aca42 bitbake: data: Drop misleading ExpansionError exception
This codepath can be triggered by a python indentation error for example.
Showing it as an ExpansionError is misleading.

Change the code to add a warning about where the failure came from (in
particular giving the variable key name that triggered it) but raise the
proper exception.

(Bitbake rev: d49d46533704e8b4404e29abfb5a7383d704c91a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:34 +00:00
Richard Purdie 2c9431135d bitbake: cooker: Drop useless parsing exception
The SyntaxError exception simply shows the recipe that failed to parse
which is pretty useless without the actual exception. We could make it
print more info, however we can just use one of the more generic handlers
instead and remove this one.

For a python indentation error, this leads to a much more readable error
message.

(Bitbake rev: 9241eb10847634e34c5ff8767ed8c114f66ff6cf)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:34 +00:00
Richard Purdie a16b543436 bitbake: data: Pass lineno/filename data from build_dependencies
If parse_python() fails, the output is confusing. Passing in the extra
file/line data isn't expensive and improves readability significantly.

(Bitbake rev: a4bb753488d322e0e31c31d6377ba780f2f824c4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:34 +00:00
Richard Purdie 958f0ff56b bitbake: codeparser: Add support for correct linenumbers
Currently, if there is something like a python indentation error in a
python function, the linenumbers and even file aren't reported correctly.

This allows lineno and filename parameters to be passed in to correct this.

The lack of a lineno parameter to python's compile() function is worked
around by using empty linefeeds. Ugly, but effective and with minimal
performance overhead.

(Bitbake rev: 5796ed550d127853808f38257f8dcc8c1cf59342)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:34 +00:00
Richard Purdie 0381b78aa4 bitbake: event/utils/methodpool: Add a cache of compiled code objects
With the addition of function line number handling, the overhead of
the compile functions is no longer negligible. We tend to compile
the same pieces of code over and over again so wrapping a cache around
this is beneficial and removes the overhead of line numbered functions.

Life cycle of a cache using a global like this is in theory problematic
although in reality unlikely to be an issue. It can be dealt with
if/as/when we deal with the other global caches.

(Bitbake rev: 98d7002d1dca4b62042e1589fd5b9b3805d57f7a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-22 00:02:05 +00:00
Richard Purdie c61c1eb26a bitbake: BBHandler: Improve IN_PYTHON_EOF handling
Now we're actively using the line numbers for other thins, having
magic values like IN_PYTHON_EOF causes problems, in particular, 32
bit overflow on 32 bit machines.

There is a neater way to signal eof to feeder(), just using an extra
parameter so use this instead and drop the IN_PYTHON_EOF magic values.

This has the added bonus that line numbers are then correct for
python functions at the end of files.

(Bitbake rev: e0f05871c2a6f1e86ae19ad343c7c6f822ddb67e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-22 00:02:05 +00:00
Ed Bartosh 5f406915b5 bitbake: toaster: remove 2 confusing parameters
Removed nobrowser and brbe script parameters as both
are confusing and nobrowser is not used anywhere.

brbe parameter usage can only be justified if toaster
doesn't work properly and user has to manually connect
toaster to running bitbake server. Even in this scenario
it's very unlikely to achieve as toaster script is not
designed for this kind of usage.

(Bitbake rev: 0fd04ede3fda6894d97a5ef830b79dbbc9c6cf51)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 13:51:54 +00:00
Ed Bartosh 3960b6ec5c bitbake: toaster: move setting of default values
Set default values of script parameters just before
they are parsed to increase readability.

(Bitbake rev: 627f0d6adcfe281ef0487bf15a35151f1ceff194)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 13:51:54 +00:00
Ed Bartosh b194c0c00a bitbake: toaster: move startup checks to a better place
Moved addr:port, bitbake.lock and toastermain.pid checks
to the place where the rest of Toaster starting happens.

(Bitbake rev: 0e7812b5512e609ea815db30be1d7caca896ab60)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 13:51:54 +00:00
Ed Bartosh 064d2c7984 bitbake: toaster: remove 2 unused functions
Removed check_pidbyfile and notify_chldexit functions from
toaster script as they're not called in the script.

(Bitbake rev: ead2823e5457c9c4d0321c2191fb99982b282d26)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 13:51:54 +00:00
Ed Bartosh c505f24ebd bitbake: toaster: remove addtoConfiguration function
This function is useless as it's called just once
and makes code less readable.

(Bitbake rev: 96e85159905e44533882ad8290b1c9dd252ff3ea)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 13:51:54 +00:00
Ed Bartosh c7e4404273 bitbake: toaster: updated header of the toaster script
Updated GPL information, years of development and
usage information. Removed outdated information about
2 ways of starting Toaster.

(Bitbake rev: b4789c9e3093e1ec8e70f67305c88709443578e6)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 13:51:54 +00:00
Ed Bartosh af34920dff bitbake: toaster: add MANAGE variable
Used MANAGE variable to avoid repeating path to
manage.py in many places in toaster script.

(Bitbake rev: e8e19168ff43b72a2481771e7a40d9d7c155cfd2)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 13:51:54 +00:00
Ed Bartosh 563b786d4b bitbake: toaster: remove unused variable
Removed unused variable RUNNING from the toaster script.

(Bitbake rev: 8b3f0b2c6efa2962d9c16e453f4d2c1af6419c6c)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 13:51:54 +00:00
Ed Bartosh aa3cc12db9 bitbake: toaster: split long lines, add/remove whitespace
Made toaster script more readable by splitting long lines and
removing and adding whitespace.

(Bitbake rev: 6d61d7903eb4fb12464bfea54d3a225f99ad394f)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 13:51:53 +00:00
Ed Bartosh 8e4acacb53 bitbake: toaster: check if address:port is in use
Used new management command checksocket to check if
Toaster can listen on address:port.

[YOCTO #8775]

(Bitbake rev: 8a306ffe5a3642fe833c875362a183096a39f641)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 13:51:53 +00:00
Ed Bartosh 847b93596d bitbake: toaster: implement checksocket command
Implemented new management command to check if it's
possible to listen on specified address:port.

[YOCTO #8775]

(Bitbake rev: 0339b90842fd7c878c511b4b89ebcaee9a431bba)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 13:51:53 +00:00
Richard Purdie 522dcaa0cb bitbake: knotty: Improve exception error message
Instead of:
"""
can only concatenate tuple (not "int") to tuple
"""
we now see:

"""
Traceback (most recent call last):
  File "/media/build1/poky/bitbake/lib/bb/ui/knotty.py", line 324, in main
    termfilter.updateFooter()
  File "/media/build1/poky/bitbake/lib/bb/ui/knotty.py", line 210, in updateFooter
    lines = 1 + int(len(content) / (self.columns + 1))
TypeError: can only concatenate tuple (not "int") to tuple
"""

which makes tacking down and fixing the problem much easier.

Also ensure we set an error exit code.

(Bitbake rev: d965bcae6cfd268406a3bd1ef77c5bb6c6e1c6d7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 12:18:19 +00:00
Richard Purdie 01d67bfa77 bitbake: knotty: Fix row/column function return value issue
When the row handling was introduced, one of the callbacks was
missed resulting in:

TypeError: can only concatenate tuple (not "int") to tuple

Fix it.

(Bitbake rev: 0b77cea2bf5b5f5704e2650fb0332f5d78037781)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 12:18:19 +00:00
Richard Purdie 6c12efa5c2 bitbake: buildinfohelper: Update for buildstats layout change
This updates buildinfo helper for the recent buildstats layout change

(Bitbake rev: 30311bbe667e9f22de17fae00ff58da06a7c3e23)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 12:18:19 +00:00
Christopher Larson 28ea1a1c95 bitbake: fetch: use orig localpath when calling orig method
When a mirror tarball is fetched, the original fetch method is called, which
unpacks the mirror tarball. After the original method is called, it checks the
localpath of the mirror tarball rather than the clone path, which isn't ideal,
particularly if the mirror tarball was removed due to being out of date. We
know the original fetch method will do what it needs to do to get its content
in the form it needs from the mirror tarball, so we can use its localpath
instead.

(Bitbake rev: 1732ad65d6c7d67b7d07cb30c074f5016adadbea)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Awais Belal <awais_belal@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 12:18:19 +00:00
Richard Purdie 5cb6d83f17 bitbake: utils: Improve traceback from better_exec internal errors
If you break the internals of better_exec(), you get a very weird
error about tb_next not being a method of None. Fix this by checking
we can step back a trace level.

(Bitbake rev: 1d710ed484f68fca0789022dde7ba877b9a894f5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 12:18:19 +00:00
Richard Purdie 0019edc818 bitbake: ast/event/utils: Improve tracebacks to include file and line numbers more correctly
Currently bitbake tracebacks can have places where the line numbers are
inaccurate and filenames may be missing. These changes start to try and
correct this.

The only way I could find to correct line numbers was to compile as a
python ast, tweak the line numbers then compile to bytecode. I'm open
to better ways of doing this if anyone knows of any.

This does mean passing a few more parameters into functions, and putting
more data into the data store about functions (i.e. their filenames
and line numbers) but the improvement in debugging is more than worthwhile).

Before:
----------------
ERROR: Execution of event handler 'run_buildstats' failed
Traceback (most recent call last):
  File "run_buildstats(e)", line 43, in run_buildstats(e=<bb.build.TaskStarted object at 0x7f7b7c57a590>)
NameError: global name 'notexist' is not defined

ERROR: Build of do_patch failed
ERROR: Traceback (most recent call last):
  File "/media/build1/poky/bitbake/lib/bb/build.py", line 560, in exec_task
    return _exec_task(fn, task, d, quieterr)
  File "/media/build1/poky/bitbake/lib/bb/build.py", line 497, in _exec_task
    event.fire(TaskStarted(task, logfn, flags, localdata), localdata)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 170, in fire
    fire_class_handlers(event, d)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 109, in fire_class_handlers
    execute_handler(name, handler, event, d)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 81, in execute_handler
    ret = handler(event)
  File "run_buildstats(e)", line 43, in run_buildstats
NameError: global name 'notexist' is not defined

----------------

After:
----------------
ERROR: Execution of event handler 'run_buildstats' failed
Traceback (most recent call last):
  File "/media/build1/poky/meta/classes/buildstats.bbclass", line 143, in run_buildstats(e=<bb.build.TaskStarted object at 0x7efe89284e10>):
         if isinstance(e, bb.build.TaskStarted):
    >        trigger = notexist
             pn = d.getVar("PN", True)
NameError: global name 'notexist' is not defined

ERROR: Build of do_package failed
ERROR: Traceback (most recent call last):
  File "/media/build1/poky/bitbake/lib/bb/build.py", line 560, in exec_task
    return _exec_task(fn, task, d, quieterr)
  File "/media/build1/poky/bitbake/lib/bb/build.py", line 497, in _exec_task
    event.fire(TaskStarted(task, logfn, flags, localdata), localdata)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 170, in fire
    fire_class_handlers(event, d)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 109, in fire_class_handlers
    execute_handler(name, handler, event, d)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 81, in execute_handler
    ret = handler(event)
  File "/media/build1/poky/meta/classes/buildstats.bbclass", line 143, in run_buildstats
    trigger = notexist
NameError: global name 'notexist' is not defined
----------------

(Bitbake rev: 1ff860960919ff6f8097138bc68de85bcb5f88b0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 12:18:19 +00:00
Richard Purdie b14ccb2367 bitbake: runqueue: Add support for <task>- syntax
It can be useful to run all tasks up to but not including a specific task. The
main reason this was never added was the lack of a good syntax. This patch
uses the syntax <taskname>- to denote this behaviour which is simple, not
invasive and fits what we need from good syntax IMO, hence we can add this.

(Bitbake rev: 99ccfd411ab3f7baa111f9f3d50fae68816a9a83)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 12:18:19 +00:00
Belen Barros Pena 03d715e541 bitbake: toaster: tables Set a default order for the software recipes table
Add default order by recipe name, so that the table content is sorted
when the page loads.

[YOCTO #8791]

(Bitbake rev: 36cc814b64bcf3825ed096ade0b8c590e497259f)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:16:13 +00:00
Ed Bartosh 4ff0d60ea8 bitbake: toaster: rework checking of Django version
Checked django version considering information from
toaster-requirements.txt, e.g. if requirements file contains
line "Django>1.8,<1.9" toaster should be able to check that
requirement correctly.

(Bitbake rev: 49976eca4a6e37e7653814c569badcd3e0fb719a)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:08 +00:00
Ed Bartosh 4a78416943 bitbake: toaster: monkey patch Queryset
Fixed 'database is locked' issue by monkey patching django QuerySet
methods.

The actual patching places were found by bisecting Django codebase.

This commit should be removed after Django is fixed if it's fixed
at all.

(Bitbake rev: 175411bf05423b1892c7928c2b928843b39645f0)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:07 +00:00
Ed Bartosh c1c8eff3f9 bitbake: toaster: removed extra calls of migrate
One manage.py migrate should be enough to apply migrations for
all applications. Extra calls just slow toaster script down.

Removed calls of migrate for orm and bldcontrol apps.

(Bitbake rev: 9299d7fa88e51294c4fd3f1354874d8253d25aff)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:07 +00:00
Ed Bartosh 507aafbd7a bitbake: toaster: work around 'database is locked' error
When sqlite can not cope with a stream of 'insert' queries it throws
'database is locked' exception.

Wrapping model.save in transaction.atomic context and repeating the call
should solve this issue.

(Bitbake rev: eb305308ca8f6228c6f52dac1bd941f29c7e5eb6)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:07 +00:00
Ed Bartosh 322b4709d8 bitbake: toaster: fixed format strings
Printing id as a number breaks toaster if object is not saved
yet as id/pk is None in this case.

Changed format spec to print id as a string in __str__
and __unicode__ methods of Build, Task and LayerVersion models.

(Bitbake rev: d5ca2bae623db214764324da5c9e4dac6beef760)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:07 +00:00
Ed Bartosh 84daa40bc7 bitbake: toaster: use OneToOneField instead of ForeignKey
Used OneToOneField to reference BuildRequest in BRBitbake model.

Fixed django warning:
WARNINGS: Setting unique=True on a ForeignKey has the same effect
          as using a OneToOneField.

(Bitbake rev: aaa4319ebbb06facb77b4ba936cf3aa2068ff238)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:07 +00:00
Elliot Smith c464f34d5b bitbake: toaster: Amend regex for MySQL database URLs
The database URL detection doesn't admit a MySQL URL without a port.

As this is a common case (e.g. you would set the HOST to a mysql.sock
path if pointing at a local MySQL server, with no port setting),
amend the regex so it will correctly recognise paths, as well
as HOST:PORT URLs.

(Bitbake rev: 89386aab888f806d5aa4a8083c06566e48d9445b)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:07 +00:00
Elliot Smith f001a4ac3c bitbake: toaster: Remove compatible_layerversions() method
Remove compatible_layerversions() and replace calls to it with
calls to get_all_compatible_layer_versions().

The sorting done in compatible_layerversions() is not relevant,
and the code can be simplified by using get_all_compatible_layer_versions().

[YOCTO #8364]

(Bitbake rev: 7bca51277314f7c0b6ee0e0d470327dfd60d37fe)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:07 +00:00
Elliot Smith 0adffdf790 bitbake: toaster: Check Django version against toaster-requirements.txt
Instead of hard-coding the required Django version in the start
script, look it up from the toaster-requirements.txt file.

[YOCTO #8364]

(Bitbake rev: 2149ff805424692f9fde29618f2ae0eb99f710d2)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:07 +00:00
Elliot Smith 8d058cfb8f bitbake: toaster: Update deprecated manage.py command
syncdb is deprecated in favour of migrate in Django 1.7:
https://docs.djangoproject.com/en/1.8/releases/1.7/#schema-migrations

Update to the "migrate" command in Toaster's start script.

[YOCTO #8364]

(Bitbake rev: 7ff1630574180e1895b90ecef1ea0caf51304446)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:07 +00:00
Elliot Smith 717c636d71 bitbake: toaster: Prevent deprecation warnings for RedirectView
The API for RedirectView is due to change in Django 1.9, which
means that Toaster generates deprecation warnings.

Set the "permanent" flag when constructing RedirectView instances
to prevent this warning.

[YOCTO #8364]

(Bitbake rev: 4aa09488bfe65cb365356b320cd9865643bb4fe5)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:07 +00:00
Elliot Smith 0f602c1a22 bitbake: toaster: Update API used to make runbuilds methods run in transactions
runbuilds previously had its methods decorated with
transaction.commit_on_success, which is now deprecated. transaction.atomic
is an (almost) drop-in replacement for this, so use this instead.

[YOCTO #8364]

(Bitbake rev: c4804b84eaaef6a81027bae5cf8bfe63d949c293)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:07 +00:00
Ed Bartosh 93f573836c bitbake: toaster: rename get_query_set -> get_queryset
Renamed get_query_set to get_queryset as get_queryset causes
deprecation warning in Django 1.7 and is deprecated in Django 1.8

(Bitbake rev: ef8b399dccd413070e32ce05e013337ea01fa64d)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:06 +00:00
Elliot Smith 23c48065d2 bitbake: toaster: Start Django machinery for database access
buildinfohelper, with the new import paths for our Django models
and Django 1.8, was not getting an active connection to the database.

In buildinfohelper, call django.setup() explicitly to make sure
that the database connection is ready and models can be queried
and saved.

[YOCTO #8364]

(Bitbake rev: 671aaab8cb7c494cd5c7621b45a6f41a203d8bb5)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:06 +00:00
Elliot Smith 7a0c45e478 bitbake: toaster: Create default project with get_or_create* method
Rather than maintain data as part of the migrations (as was
done for the default project previously), create the default
(cli builds) project on demand as a by-product of getting
it from the database.

[YOCTO #8364]

(Bitbake rev: 5fd8e90ab9b81d1bd0d301bc1c91228ecbbea74b)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:06 +00:00
Ed Bartosh 9de8dfa11a bitbake: toaster: Fix references to app paths
The buildinfohelper runs outside of Django, but still needs
access to our Django app classes.

Previously, the imports referenced the toaster.* app, which worked
fine. But in Django 1.8, this causes an error about the same
module being loaded multiple times from different paths.

Change the paths to our Django modules so they don't cause
this error to be thrown. We can do this as we've added our
application libraries to sys.path in the buildinfohelper anyway.

[YOCTO #8364]

(Bitbake rev: 070da64cf32c32b5ffc34d611b463c3a3960b419)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:06 +00:00
Ed Bartosh 535fc9b6fb bitbake: toaster: Remove South migrations
South has been deprecated in favour of Django's own migration framework,
so remove the old South migrations and replace them with Django ones.

[YOCTO #8364]

(Bitbake rev: 427d8bc02d1aa00a19057602d592d58334514804)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:06 +00:00
Elliot Smith 8ca46641ef bitbake: toaster: Upgrade to Django 1.8.6 and remove South
Upgrade Django to long-term support version.

Django now provides its own migration framework, so remove
requirement for South.

[YOCTO #8364]

(Bitbake rev: 648b62654c52116451c6a68a46d7264db3a34d09)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:06 +00:00
Ed Bartosh b322dec016 bitbake: toasterui: process SetBRBE event
Currently BRBE is known to toasterui only when build is
started. It's passed to it with BuildStarted event. This is
too late as if build fails earilier than build starts toasterui
can not inform Toaster about the failure.

Set BRBE as soon as it's provided by Toaster.
This should make toasterui to be able to inform Toaster
about early build failures, e.g. failures during recipe parsing.

(Bitbake rev: d7819508dac488a64be3caec88db285cda9599ab)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:06 +00:00
Ed Bartosh 0274b68450 bitbake: toaster: trigger SetBRBE event
Triggered MetadataEvent "SetBRBE" when TOASTER_BRBE variable
is set on bitbake server. This should make buildinfohelper
aware of the build request id, which is used to properly
report build status and failures back to Toaster.

(Bitbake rev: 8b136fb093020bc912a7b21d5163e1cd5fb12124)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:06 +00:00
Ed Bartosh fdb8e7433c bitbake: toaster: implement BitbakeController.triggerEvent
This API will be used to pass data from Toaster to
toaster_ui and buildinfohelper.

(Bitbake rev: 6dca6fd47c5e8d66687fb07177e98cd531cd971d)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:06 +00:00
Richard Purdie 5de3800c30 bitbake: event: Fix subprocess event error traceback failures
If subprocess raises a CalledProcessError() error, e.g. from a call
like subprocess.check_call("false"), bitbake would try and pass the
object over IPC and fail, leading to an unusual error:

('__init__() takes at least 3 arguments (1 given)', <class 'subprocess.CalledProcessError'>, ())%

To avoid this, we turn the value into a string which prevents the
issues the IPC has trying to deal with the object (for the same reason
we deal with tracebacks here too).

[YOCTO #8752]

(Bitbake rev: 05695424b918fc81b16cbac70d79d8271a0b6045)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:06 +00:00
brian avery 7e8ff7b9d7 bitbake: toaster: toasterui Add ParseStarted/ParseProgress events to mask
Toaster is not able to see ParseStarted and ParseProgress events
for command-line builds. This means it's not possible for Toaster
to detect failed builds, if the failure occurs at a point before
the BuildStarted event, as the build won't show up at all.

Add these events to the event mask, so that Toaster's toasterui
can detect and respond to them.

(Bitbake rev: 16bfd3e3d145705a2b3a05648ddbcacc7a338dfa)

Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-09 09:03:35 +00:00
Richard Purdie f9384b0669 bitbake: knotty: Enforce terminal line limit to stop crazy scrolling
If there are more tasks running than there are lines on the terminal, the data
scrolls in ways the UI wasn't designed for. This patch adjusts the UI just to
show the processes which fit onto the number of rows in the terminal window.
You can see the total number running from the counter in the top left as usual
and this makes warning and errors messages scrolling from the top of the window
work as designed.

Ultimately, scrolling would be nice but is for another time, this fixes the
biggest UI issue on highly parallel machines.

(Bitbake rev: 67b77658e2bfa849f6f55c9c262cb11d6bfdb399)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-08 10:31:48 +00:00
Alexander Kanavin 3745479596 bitbake: bitbake: rename REGEX, REGEX_URI, and GITTAGREGEX.
Rename REGEX to UPSTREAM_CHECK_REGEX, REGEX_URI to UPSTREAM_CHECK_URI, and
GITTAGREGEX to UPSTREAM_CHECK_GITTAGREGEX to better reflect their purpose
and to reflect a common namespace.

(Bitbake rev: f0a9e783f9969573fd74edfa241ef14f18ac684e)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-08 10:20:31 +00:00
Ed Bartosh dd282d475c bitbake: toaster: return back 'New project' button
The button was removed by mistake in merging modes patchset.

(Bitbake rev: f7b33a95b5d53973bc673373bdd2652f4d32f0c2)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:39:08 +00:00
Mihail Stanciu 2a8e970d42 bitbake: toaster: tests Update UI tests to work with 2.0 changes
Update tests that were referencing items removed in 2.0.

Update most xpath elements left to use IDs.

(Bitbake rev: c4dda67dd3773d02b760d96dd9c6f26bff93533d)

Signed-off-by: Mihail Stanciu <stanciux.mihail@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:39:08 +00:00
Mihail Stanciu fe8a0a3697 bitbake: toaster: tests Automated build-mode backend tests
Add tests to check various backend scenarios that
require UI interaction.

Add necessary files for above tests (the sqlite file
and the default config json).

(Bitbake rev: 8dcad75ad1d9b8adddeca57e996b8d904b209df5)

Signed-off-by: Mihail Stanciu <stanciux.mihail@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:39:08 +00:00
Ed Bartosh 0497b570e5 bitbake: toaster: unset environment variables
Toaster script sets environment variables when it starts.
It makes sense to unset them when toaster stops as they can
cause other programs to behave incorrectly.

(Bitbake rev: 7a2e9745ff3ef6d6360f74b222248d4bd35e6a08)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:11 +00:00
Ed Bartosh 8b7a548e30 bitbake: toaster: get rid of complicated heuristics
Removed buildinfohelper code which was trying to guess layer version of
the recipe using build request information. The code caused creation of
duplicated recipes as it resulted in layer version from layer index
instead of returning build layer version. As a result of this Toaster
UI was not showing any information about recipes.

Default approach used to find layer version seems to work much better as
it finds proper layer version. Now toaster will use it as the only
way to find layer version.

(Bitbake rev: 101690bda7ad55dc0657483233c90c374713755b)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:11 +00:00
Ed Bartosh 556b8b6156 bitbake: toaster: remove SDKMACHINE from project variables
Removed SDKMACHINE from predefined set of variables
for the project as it causes bitbake build error:
   SDKMACHINE is set, but SDK_ARCH has not been changed as a result

This variable does not need to be predefined as it's not used by
toaster.

It's still possible to specify it in project configuration
if needed. SDK_ARCH variable should be set too to avoid above
mentioned build failure.

(Bitbake rev: a6ab2a38ab666376b4a5e796d3a88929e8cbfb22)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:11 +00:00
Ed Bartosh 4186f5b04e bitbake: toaster: stop using toaster-pre.conf
This file is not needed anymore as toaster UI doesn't
write variables to it. It sets them directly on bitbake
server.

(Bitbake rev: 624240460ec32af918dcea8c04a913e3b94e012b)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:10 +00:00
Ed Bartosh 361faa3e6c bitbake: toaster: remove writeConfFile API
This API is not used anymore as toaster doesn't write variables
to configuration files anymore. It sets variables through its
connection to bitbake server.

(Bitbake rev: 6dda2916b75d688874d208192d5c7cdb302eec35)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:10 +00:00
Ed Bartosh fcbba5a89b bitbake: toaster: set varibales on bitbake server
Set project variables using connection to bitbake server
instead of writing them to toaster configuration files.

This is a way to avoid rewriting configuration files before
and after every build, triggered by toaster UI.

(Bitbake rev: 3f77c69b867735b5d27c6ab7c62e39ac8d5cb2b5)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:10 +00:00
Ed Bartosh 993bc7eacd bitbake: toaster: implement BitbakeController.getVariable
Added new API to get value of bitbake variable from bitbake server.
The API will be used to update INHERIT variable instead of writing
it to the toaster configuration files.

(Bitbake rev: effc371e3a4a2262e53e9e560c7c08cffad195da)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:10 +00:00
Michael Wood 53e981e583 bitbake: toaster: buildinfohelper Broaden the toaster created recipe data case
When build information is the 'original' source of the information we
need to return the recipe that was created rather than the copy of the
recipe that is taken for keeping build history. We do this already for
command line triggered builds, but we also have this case for custom
images. We can simply check if the built_recipe exists instead of
special casing this.

(Bitbake rev: 9a8653bf602b2111dee7ee6a459682a68a695b22)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:10 +00:00
Ed Bartosh 57e5f24e57 bitbake: toaster: do not create duplicate HelpText objects
buildinfohelper code expects only one HelpText object per
build/variable/description.
Current code creates more than one such an object, which causes
toastergui to crash with this exception:
 MultipleObjectsReturned: get() returned more than one HelpText -- it returned 2!

Used git_or_create API to ensure that only one HelpText object is
created.

(Bitbake rev: e9b46803eb6f1f4044919abf90c8aeb3536e73ed)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:10 +00:00
Ed Bartosh 4c1e5ec29a bitbake: toaster: remove usage of BUILD_MODE variable
As both modes can be used at the same time we can't have any
difference in UI between modes.

Removed all conditional statements that used BUILD_MODE.

(Bitbake rev: 4ce1559c2558bd0fd278ff02a1a93bec03c4156b)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:10 +00:00
Ed Bartosh 990289511b bitbake: toaster: do not terminate bb server
Toaster needs bb server to be running all the time due
to merged analysis and managed modes. Server gets restarted
before every build triggered by UI, but it shouldn't be
terminated as it will influence command line builds.

[YOCTO #8279]

(Bitbake rev: e69c87c7842e796ffcd7193ecde22c8f688498f5)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:09 +00:00
Ed Bartosh 58765a8719 bitbake: toaster: remove stopBBServer API
Removed stopBBServer API from build controller as toaster
doesn't stop bitbake server anymore. It's reused for both
types of builds: triggered by UI and started manually.

(Bitbake rev: 0cad802da171d51814f22cc9383f496a63cd0c75)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:09 +00:00
Ed Bartosh 95a3cf7549 bitbake: toaster: reimplemented startBBServer method
Rewritten LocalhostBEController.startBBServer to use
'toaster restart-bitbake' and read bitbake port number from
bitbake.lock.

Removed complicated logic of running oe-init-memres and looking for
bitbake port number in the logs.

(Bitbake rev: 4b065353c3454923a1ef88e9f0a8702e5626060e)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:09 +00:00
Ed Bartosh 76d53b5856 bitbake: toaster: remove _setupBE function
This function is not needed as build environment is always created
because of the new way to run Toaster. It can be only sourced after
oe-init-build-env is sourced.

(Bitbake rev: 27a59b4373a449f17e46ab93e0eba15cf75967ae)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:09 +00:00
Ed Bartosh 87b2f954b9 bitbake: toaster: implement 'toaster restart-bitbake'
This implementation allows to have functionality of restarting
bitbake in toaster script. It can be used by toaster script and
build controllers.

[YOCTO #8279]

(Bitbake rev: 24cd26b39014419d883b7cf45e4943c301d585cc)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:09 +00:00
Ed Bartosh 891484ada9 bitbake: toaster: implement start_bitbake function
Separated functionality of starting bitbake server and observer
processes.

This functionality will be used by build controllers to restart
bitbake processes.

(Bitbake rev: d8e9ce7ba3f5b89c51e769bd4a46c2026a110d40)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:09 +00:00
Ed Bartosh bf25471341 bitbake: toaster: implement stop_bitbake function
Separated functionality of stopping bitbake server and observer
processes.

This functionality will be used by build controllers to restart
bitbake processes.

(Bitbake rev: dd40d70b707a3ff59edfa69923d4d9b671928a66)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:08 +00:00
Ed Bartosh 7c2b22556b bitbake: toaster: update brbe and project attributes
Updated attributes of buildinfohelper object as they can
be changed for every build. For example brbe is set by
runbuilds for every build triggered by Toaster UI.

(Bitbake rev: ea3bc8d01704dc64f6cb7b4f5fe66c312a575174)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:08 +00:00
Ed Bartosh de812d086e bitbake: toaster: start 'manage.py runbuilds' in the script
Running runbuilds machinery in background allows Toaster to
start builds initiated by toaster UI.

Added runbuilds pid file to the list in webserverKillAll to
kill runbuilds the same way as runserver process.

(Bitbake rev: 91bf323e873c31d883b01ea993c005f45382e83f)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:08 +00:00
Ed Bartosh 28e8ccf437 bitbake: toaster: make runbuilds to loop
Made runbuilds to loop to avoid having a loop in shell code and
initialize heavy Django init machinery every second.

Ignored exceptions to prevent exiting a loop.

(Bitbake rev: e04da15556ca0936de652b8c085e4199e5551457)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:08 +00:00
Ed Bartosh a3871a37de bitbake: toaster: use parent of the build dir
Assigned TOASTER_DIR to the parent of the build directory.
This should fix local controller crash as it assumes that
TOASTER_DIR is a root of local poky and tries to clone it.

(Bitbake rev: 124eb2bb15de18e4d2d6483c3549e1d887234736)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:08 +00:00
Ed Bartosh 2a96d35893 bitbake: toaster: check for toaster configuration later
Moved check for toasterconf.json after check of build environment.
We'll need some variables from build environment to find toasterconf.json
better way.

(Bitbake rev: a4f6809237dffb1abff78c6a8c82bcc4852aedfa)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:08 +00:00
Ed Bartosh d87a5346cf bitbake: toaster: remove unused variable
Removed variable RUNNING as it's not used anymore
in the toaster script.

(Bitbake rev: 187c31a8789c37723b9288afd35f1785f1993895)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:07 +00:00
brian avery dc6a489c65 bitbake: toaster: change toasterconf.json logic to use TEMPLATECONF, like oe-setup-builddir
This changes the way we pick a toasterconf.json file so that it
matches the way oe-setup-builddir does it. We now base it on TEMPLATECONF
found in the .templateconf file.

(Bitbake rev: 8476db82ea80edf2b70bc2082bcecee09037556d)

Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:07 +00:00
Ed Bartosh 5a42c2d050 bitbake: toaster: run bitbake the same way
Run bitbake in toaster script the same way as it's run
by localbuildcontroller.

(Bitbake rev: c5d5da095d5eace03f8a50910f5a60ab35acbb1a)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:07 +00:00
Ed Bartosh cac91db01d bitbake: toaster: set DATABASE_URL in toaster script
Set environment variable DATABASE_URL in bitbake/bit/toaster
Variable value is obtained by running 'manage.py get-dburl'

(Bitbake rev: ac4800cc8b765b270f27ea9a5938f3f2a2e0ced4)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:07 +00:00
Ed Bartosh a464bf23b4 bitbake: toaster: implement get-dburl command
Implemented management command to call getDATABASE_URL API.
It will be used to get database url from toaster shell script
by running 'manage.py get-dburl'

(Bitbake rev: fabe9c8f14fd7b8ab204a2b610c64ac5b62135ac)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:07 +00:00
Ed Bartosh e4731514f8 bitbake: toaster: don't allow to run toaster as a script
Removed support of starting toaster as a script.

Sourcing a toaster script becomes the only way to start it.
It's consistent with the way oe build system is started by sourcing
oe-init-build-env. It also returns user back to shell, so user can
continue running builds without having to open new terminal window.

[YOCTO #8279]

(Bitbake rev: 7e8632c0527245c1474e487dd9a972948e794471)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:07 +00:00
Paul Eggleton 4de214f752 bitbake: lib/bb/utils: improve edit_bblayers_conf() handling of bblayers.conf formatting
Make the following improvements to edit_bblayers_conf():

* Support ~ in BBLAYERS entries
* Handle where BBLAYERS items are added over multiple lines with +=
  instead of one single long item

Also add some comments documenting the function arguments and return
values as well as a set of bitbake-selftest tests.

(This function is used by the bitbake-layers add, remove and
layerindex-fetch subcommands, as well as devtool when adding the
workspace layer).

(Bitbake rev: e9a0858023c7671e30cc8ebb08496304b7f26b31)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:01:21 +00:00
Paul Eggleton 0debb11883 bitbake: lib/bb/utils: fix error in edit_metadata() when deleting first line
If you tried to delete the variable on the first line passed to
edit_metadata() this failed because the logic for trimming extra blank
lines didn't expect the list to be empty at that point - fix that bad
assumption.

(Bitbake rev: 8bce6fefdc5c046b916588962a2b429c0f648133)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:01:21 +00:00
Alexander Kanavin 9d19dd9bd7 bitbake: wget.py: parse only <a> tags
For two reasons:
1) The important one: we hit the following bug when doing upstream version checks
on some webpages:
https://bugs.launchpad.net/beautifulsoup/+bug/1471755

2) Also, documentation for beautifulsoup states that memory usage and
speed is improved that way.

(Bitbake rev: 7546d4aeb3ba8fda9832081b84d93138dc5e58d6)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:01:21 +00:00
Michael Wood 71ede7b689 bitbake: toaster: toastergui tests Add generic test for ToasterTables widget
For each of the ToasterTables defined test:
 - Data returned
 - Search
 - Order by
 - Limit
 - Paginate
 - Filter

These tests test that the functions on all tables work, it does not
validate the content of the data in the tables themselves, this needs
to be done in table specific tests.

(Bitbake rev: 649d563f95ffc57d0fe3dbf494e7dbffcc99a1b4)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:01:20 +00:00
Michael Wood 34b22cf897 bitbake: toaster: tables Fix invalid field name on NewCustomImagesTable
Correct the field name in the NewCustomImagesTable as it is a Recipe
object it's self and not a container.

(Bitbake rev: ebd1c493d8b7fb9ee7b3e40c17165dc9c22ca795)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:01:20 +00:00
Michael Wood 1c59846fc0 bitbake: toaster: tables Add default_orderby field where it was missing or unset
This value is used to set the default ordering of the model that is used
for ToasterTables, it is picked up client side to set the ordering
indicator.

[YOCTO #8695]

(Bitbake rev: f1c91bff810a579b169c46b7710e22f5553b484e)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:01:20 +00:00
Michael Wood d82c541147 bitbake: toaster: CustomImageRecipe add search_allowed_fields to this model
In order to search the model from the UI some fields must be nominated
as searchable.

(Bitbake rev: 2558729b4c248a2fa5a11d877bc42cb05a30602e)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:01:20 +00:00
Michael Wood bdf62414f2 bitbake: toaster: machines table Fix missing layers information needed for filter
The current layers information wasn't being passed to the template for
the Select/Add button for the Compatible machines filter.

(Bitbake rev: 559102984d3f62a7675899e058166bdf4d552c46)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:01:20 +00:00
Michael Wood b90a8dca99 bitbake: toaster: tablejs Make sure click handlers consume click event
Avoid the click event from propagating and causing strange side effects
in toaster tables.

[YOCTO #8527]
[YOCTO #8148]

(Bitbake rev: f23e54292a673b2fd3f17d317c21984c753727c7)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:01:20 +00:00
Michael Wood c075bcf545 bitbake: toaster: projectpage Make sure build targets are space separated
Make sure the build targets are space separated when building multiple
targets. Also fix error path now that YOCTO #7995 is resolved.

[YOCTO #8450]

(Bitbake rev: e9719eb2e8249f0d10b39bfdd4aef563368b5ffd)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:01:20 +00:00
Belen Barros Pena 88f9310ce0 bitbake: toaster: builds pages Fix the download cooker log link
The 'all builds' page was missing the download icon next to the outcome
icon, which allows you to download a build log from the 'all builds' page.

This patch brings it back. It also adds a check in the project builds page
to make sure the download icon only appears if the build generates a cooker
log, since builds that fail at the build request stage do not generate a
cooker log.

(Bitbake rev: 50d7ffa697764d84d96f26a3ae58a9dacde7b31e)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-11-16 11:55:08 +00:00
Belen Barros Pena d04af8b571 bitbake: toaster: project pages Link to image recipes table in notifications
The project created notifications and the 'Choosea recipe to build' link in
the project page are linking to the software recipes table. Changes them to
link to the image recipes table instead, which is bound to be more useful.

(Bitbake rev: 10e1fbee0d79dbbf15b89ebc155bb2c0092a71f2)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-11-16 11:55:08 +00:00
Ke Zou 70465c717a bitbake: toaster: tests: Re-write some cases to make them more maintainable
(Bitbake rev: c3c7e94cadb3a3a2d8ef33ed456f439c106018c1)

Signed-off-by: Ke Zou <ke.zou@windriver.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-11-16 11:55:08 +00:00
Richard Purdie 536b73f55f bitbake: data_smart: Only support lowercase OVERRIDES
Our current OVERRIDES handling means we end up caching and checking for
a lot of possible override combinations which turn out to very unlikely.
A typical example is the SRC_URI variable where we have to check if
"URI" is an override. Having spent many hours working in this code, I've
realised all the actual overrides we use are lower case and our standard
variables are mostly uppercase.

This means we could gain quite some speed advantage if we write this
into the code, that overrides only consist of lowercase characters. This
patch shows how simple this is and the resulting speed gains are
significant. This is a significant change but tests show we don't appear
to have any users of capitals in overrides in any OE-Core metadata.

Before "time bitbake -p":

real	2m4.224s
user	7m32.312s
sys	0m7.116s

After "time bitbake -p":

real	1m26.009s
user	5m10.484s
sys	0m4.640s

This check could also be made conditional however I'm not seeing a need
to do that at present.

(Bitbake rev: c9b9443faa76ee7366b1400a56f826f3f9dec1be)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-11-16 11:55:08 +00:00
Richard Purdie fb01a66f48 bitbake: fetch2: Remove crazy code in unpack
This looks reasonable until you realise self.localpath is a function. Data
expansion of something which isn't a string is the original value so this
code just wastes CPU cycles and makes no sense. Remove it.

(Bitbake rev: 37214ea9bf484998b75dbc1200d53f1afc5257ed)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-11-16 11:55:08 +00:00
Richard Purdie 7db88aa67d bitbake: parse: Don't try to expand __base_depends/__depends
Trying to expand a variable which isn't a string doesn't make sense.

(Bitbake rev: 62367cca1f1793eb9827406bcdd5980fdeb80a60)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-11-16 11:55:07 +00:00
Richard Purdie 4c04ce0576 bitbake: cache: Don't try to expand __inherit_data
Trying to expand a variable which isn't a string doesn't make sense.

(Bitbake rev: 54d0ddd166a6707b4f8c8535639e3055b783363b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-11-16 11:55:07 +00:00
Elliot Smith 9d8e36a6c9 bitbake: toaster: localhostbectrl Pass DATABASE_URL in via the process environment
Instead of putting the DATABASE_URL as part of the command for launching
the bitbake observer process set it as part of environment.

This fixes two issues 1. Where the value isn't quoted and therefore will be
interpreted in the shell and 2. Anyone being able to see the value of
DATABASE_URL in the process tree.

[YOCTO #8669]

(Bitbake rev: 832a8523067606b180c02f0d1544e8a23219bb08)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-11-16 11:55:07 +00:00
Michael Wood 4677d8b4fb bitbake: toaster: Remove the new-build-input button widget
The button required a lot of state maintenance to make sure it
showed up when the project was configured properly, showed correctly
according to the projects known to Toaster, displayed correctly
according to the mode Toaster was in, and was able to be
used to change the current project.

(Bitbake rev: b644514a96f3947ad3f307a26301c064c8ae18f8)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-11-16 11:55:07 +00:00
Michael Wood 55f44943d7 bitbake: toaster: projecttopbar Use the project in context to get num builds
Use the project in the template context rather than ajax request to get
the number of ended builds.

(Bitbake rev: 1f6bfa876b17a600fc83c789af8aaa2639a463b2)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-11-16 11:55:07 +00:00
Michael Wood e9d4962a0c bitbake: toaster: projectpage Disable/Enable build input if we have 0 layers
If we've removed all the layers in the configuration, disable the build
button and build input.

(Bitbake rev: 872bd5ccf58236f5146b1640cc1c465b58371e44)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-11-16 11:55:07 +00:00
Michael Wood 5fa4c73603 bitbake: toaster: orm Fix get_number_of_builds to count all apart from IN_PROGRESS
The count of a project's builds should not include those which are
currently in progress.

(Bitbake rev: a981700701c41c7bbb6a9778e95f691278c5c294)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-11-16 11:55:06 +00:00
Richard Purdie c4032f479e bitbake: codeparser: Only load the codeparser cache once
The server state gets reset multiple times during startup and currently
we reload the codeparser cache each time. This is pointless and causes
unnecessary interaction time with bitbake.

(Bitbake rev: 4f700316933e8e7b2d27366e5ce6176895b913e7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-11-16 11:55:06 +00:00
Elliot Smith a0791c19db bitbake: toasterui: Create per-build logs
Rather than relying on bug 8411, which is conveniently creating
separate log files for each of our builds, create our own
log file for each build.

The log files are created in the same tmp directories that
bitbake users, but are timestamped to the millisecond to avoid
name collisions.

Each log file is opened on a ParseStarted event (for builds
triggered by Toaster) or BuildStarted event (for builds on the
command line: Toaster doesn't get the ParseStarted event
for command-line builds).

The log file is closed on the BuildCompleted event, or if the
build fails.

Because we start logging on ParseStarted for Toaster builds,
we're able to capture the "Build Configuration" section which
bitbake writes to output.

[YOCTO #8373]

(Bitbake rev: 7974203cd8bc66dff1fcc55f8723dedefaf72840)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-29 22:05:36 +00:00
Richard Purdie 290534d3ff bitbake: build/utils: Add BB_TASK_IONICE_LEVEL support
Similarly to BB_TASK_NICE_LEVEL, add BB_TASK_IONICE_LEVEL which allows the ioprio
of tasks to be adjusted. This is in response to various qemu runtime timeouts
which have been witnessed on the autobuilder, seemingly due to IO starvation (we
already use NICE_LEVEL to adjust tasks). This has a fairly urgent need to deal
with certain 'random' failures we're seeing on the autobuilders in testing.

The format of the data in the variable is BB_TASK_IONICE_LEVEL = "<class>.<prio>".

For <class>, 2 is best effort (the default), 1 is real time and 3 is idle. You'd
need superuser privileges to use realtime. The <prio> value is a default of 4,
and can be set between 0 and 7 with 7 being lowest priority and 0 the highest.
The user can set this freely with normal privileges

Note that in order for this to take effect, you need the cfq scheduler selected
for the backing block device.

We could use nice wrapper functions for ioprio from modules like psutil however
that would complicate bitbake dependencies. This version has some magic numbers
but works on the main 32 and 64 bit x86 build architectures and can easily be
extended if ever needed. When we move to python 3.x, we can likely replace this
with standard calls.

(Bitbake rev: b9471ad147b102c45d65f5ffd9521864df7ff9c1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-29 07:35:52 +00:00
Richard Purdie 3ebf7617d6 bitbake: cooker: Ensure BB_CONSOLE remains correct over server resets
The console log data is written to is created at console initialisation
time and does not change over reset events. This ensures the
BB_CONSOLELOG value is correct over such resets by preserving it.

(Bitbake rev: 335eb2db228f7543a49de71f063ac72b865c947a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-29 07:35:52 +00:00
Richard Purdie 5b19b71252 bitbake: bb/ui: Use getSetVariable command for BB_CONSOLELOG
Metadata can define BB_CONSOLELOG as containing ${DATETIME} and
this can get expanded to a different value each time the variable
is read. In the case of BB_CONSOLELOG, this behaviour is not
desireable.

The values of DATE/TIME are locked down at build time but this is too
late for the purposes of ensuring the system can figure out the real
value of BB_CONSOLELOG.

The best way to do this is to set the variable into the datastore, thereby
preserving its value.

[YOCTO #8411]

(Bitbake rev: 021f2eb55ab5863b57ed1b3f19f1b329bc1ad477)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-29 07:35:51 +00:00
Richard Purdie acc7b4d931 bitbake: command: Add getSetVariable command
There are some use cases where we want to read a variable but also
set the variable to the value read, effectively locking in any
expansion of it. This adds such a command.

(Bitbake rev: 0c0c524691e3d2ffd9953a106fcc06262cbde910)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-29 07:35:51 +00:00
Scott Rifenbark c8051c52a4 bitbake: bitbake-user-manual: Added new description for BB_TASK_IONICE_LEVEL
Added a new variable description here for this variable.

(Bitbake rev: 4e0da1ef4a58a7ba4b92d4e07ef79cf7bd90abf0)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-29 07:35:51 +00:00
Scott Rifenbark 183290a9d4 bitbake: bitbake-user-manual: Added BBTARGETS variable description.
(Bitbake rev: 0f0c205b9d6c24bee334eabb359973e0368ce931)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-29 07:35:51 +00:00
Belen Barros Pena 66d3c353f8 bitbake: toaster: templates Add meaningful title tags
Our title tags are all over the place, and have no relation to the page
content. This commit adds a meaningful title tag to each Toaster page.

(Bitbake rev: 1ab8827d684a19a70f3b788aed2327bf30edffe2)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-29 07:32:10 +00:00
Michael Wood 62e200e3e3 bitbake: toaster: orm Fix restrictive LogMessage message length
Log messages can be quite long so use a TextField rather than a char
field with max length of 240. mySQL is especially picky about field
lengths and will cause an exception if the log is too long.

[YOCTO #8475]

(Bitbake rev: 349119f44cdad240dda7f4db45992e2539e1416f)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-27 07:24:31 +00:00
Elliot Smith 78f935df8f bitbake: toaster: Remove all navigation when not in build mode
The user is redirected to the all builds page or all projects
page from the landing page, regardless of mode.

In build mode, this makes sense; but in analysis mode, we are
restricting the view to just the cli builds project. This means
that "all projects" and "all builds" only contains items relating
to this one project.

Modify the landing page so it redirects to the project builds page
for the cli builds project when not in build mode. Also remove
navigation elements which are irrelevant when not in build mode.

[YOCTO #8514]

(Bitbake rev: ae754655fa1bc5168f43e8821773e7b7b9440a5d)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-27 07:24:31 +00:00
Elliot Smith c5f147b4ab bitbake: toaster: Run tests in build mode
Now that the UI content is conditional on whether BUILD_MODE
is active, modify the existing tests so that they run in
this mode by default.

[YOCTO #8514]

(Bitbake rev: 991712f3f88759e2d68af5a3fe83c1395c4f67f6)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-27 07:24:31 +00:00
Elliot Smith 1d17109d62 bitbake: toaster: Hide builds for non-cli projects in analysis mode
The "latest builds" sections of the "all builds" page
show builds for all projects. This is not appropriate
for analysis mode, where we can only affect the command-line
builds project.

Modify the "latest builds" section to only show builds for
the command line builds project if in analysis mode.

Also rationalise where we get the queryset of latest builds from:
we have a _get_latest_builds() function which was being used
to get the latest builds in most places, but not all.

Also modify _get_latest_builds() to sort by started_on, rather
than primary key, as assuming that a higher primary key value equates
with later start time is incorrect.

[YOCTO #8514]

(Bitbake rev: 52de2c8cd11fa743f8fd7cb0cf776eea622ac474)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-27 07:24:31 +00:00
Elliot Smith a580479452 bitbake: toaster: Hide top bar buttons in analysis mode
The "new build" and "new project" buttons are irrelevant in
analysis mode, as you can't start a build or a project.

Hide these buttons if not in BUILD_MODE.

[YOCTO #8514]

(Bitbake rev: b1858653c22eb6d51fc23f75ccad69b76523eb41)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-27 07:24:31 +00:00
Elliot Smith 1ec2ec3429 bitbake: toaster: Show mode-appropriate landing page
The same landing page is shown for both analysis and
build modes. This means that users in analysis mode can see
options which are not available or broken in that mode.

Modify the landing page template to show a simple "run a build"
message if the user is in analysis mode and has no builds yet.

Also clean up the landing page HTML, because the indentation was
a mess and the HTML was invalid.

[YOCTO #8514]

(Bitbake rev: 86b4a10cc817abb5e3e63360dc8b3c550337f17c)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-27 07:24:31 +00:00
Elliot Smith bbac0f092a bitbake: toaster: Add BUILD_MODE flag to context
We set a TOASTER_MANAGED env variable in the startup script,
which has a value of "1" if Toaster should run in build mode.

Add a BUILD_MODE variable to settings.py which is True if
TOASTER_MANAGED is set to "1", False otherwise.

Add this to the context for every template, so we can use this
information to conditionally alter the content of pages
according to the mode we're in.

[YOCTO #8514]

(Bitbake rev: f9cf4739fc85c9760ce748323dc7c8de3fa7eaec)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-27 07:24:31 +00:00
Ed Bartosh 851f0d82d9 bitbake: toaster: add get_or_create_targets API
Target objects are created before the build if build is
started from UI in build mode. However, in analysis mode Target
objects don't exist and need to be created using information
from bitbake events.

Added new API call get_or_create_targets to retrive existing
target objects or create them if they don't exist yet.

(Bitbake rev: ef69be31d133696bde54605f5a18da660099734c)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-27 07:24:30 +00:00
Jens Rehsack dcd9cd0f9e bitbake: fetcher: svn: Add support for checkout to a custom path
Add support for the Subversion fetcher to checkout modules to a custom path
than the module name to avoid checkout is always module - svn is path
based and tag/branch-checkout might break builds because of invaid path specs.

(Bitbake rev: af88f538e61afa1b115be4d7afe00d8477f61750)

Signed-off-by: Jens Rehsack <sno@netbsd.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-27 07:24:30 +00:00
Ed Bartosh 4ab7202c31 bitbake: cooker: preserve pre and post configs
Additional config files passed to bitbake server with --read and
--postread options are rewritten by client bitbake even if
it doesn't use those options.

This is a show stopper for toaster as toaster command line
builds are based on the assumption that server is aware of
toster configs, provided by --postread option.

This behaviour is fixed by preserving values of --read and
--postread options when bitbake server starts and restoring
them if client bitbake doesn't explicitly specify them.

(Bitbake rev: 02c64f7487ca8ec5d32c440f5002c4b8f64b76a6)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-27 07:24:30 +00:00
Elliot Smith a9053ac082 bitbake: toaster: Add tests for error message display on the build dashboard
Add tests to ensure that log messages with CRITICAL level and
with EXCEPTION level are shown in the main errors display
on the build dashboard.

[YOCTO #8320]

(Bitbake rev: 7b3ed62800cb7352154897c1c5ad2440a54df198)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 15:00:00 +01:00
Elliot Smith 2517987f8d bitbake: toaster: Modify "New build" button behaviour for cli builds project
The "New build" button should only be displayed if there are
user-generated projects, not if there is only the command-line
builds project. (Toaster can't run builds on behalf of the
command-line builds project.)

The "New build" form should also display as if no project has
been set (i.e. show the project and recipe text entries).

Add a variable which tracks the number of non-command-line projects,
then use this to hide the button when appropriate.

Also track whether the current project is the default one, and
use this to add extra conditions to when the "New build" text
entries are shown.

[YOCTO #8231]

(Bitbake rev: 07e7bc29a7d976941eeae997f78a25ed6edddfc8)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:59:59 +01:00
Elliot Smith 56d4c84cec bitbake: toaster: Clean up template code
Some errors in the HTML template didn't show up until
they were rendering command line builds.

One useless conditional was lingering in another template.

Clean up these issues to display the latest builds section
correctly.

(Bitbake rev: c781d6be94a25f942946d3a406e07791385e4596)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:59:59 +01:00
Elliot Smith d96cedf51f bitbake: toaster: More linting of tests
Fix some more lint errors on the tests for toastergui.

(Bitbake rev: a31710db562a91da8ef37f5cd2231132474f46da)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:59:59 +01:00
Elliot Smith 7c8877efde bitbake: toaster: Show tooltip next to cli builds project name in all builds
In the all builds page, show an icon with tooltip next to the
command line builds project name.

[YOCTO #8231]

(Bitbake rev: 0240d92c6a508e0b22f37b3ea804171dd3dea5a6)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:59:59 +01:00
Elliot Smith 76702347a8 bitbake: toaster: Hide tabs and add info popups for command line builds
Command line builds don't have a configuration or
layers which can be modified through Toaster.

Change the project builds page for the command line builds project,
to hide the tabs and add some info popups in appropriate places on
that page.

[YOCTO #8231]

(Bitbake rev: 565611749d47c915035890db60d19ab2fca7c42e)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:59:59 +01:00
Elliot Smith da4c6144f1 bitbake: toaster: Make the builds view the project page for "command line builds"
Command line builds don't have configuration or layers which can
be manipulated in Toaster, so these pages shouldn't be visible.
However, the configuration page is the default page for the
project view (/project/X/), which isn't correct for the
command line builds project.

Modify all project page links across the application so that
the command line builds project (aka the "default" project)
always displays the builds tab.

Add a project_url tag for templates which contains the logic
determining where the URL for a project links to, based on
whether it is the default project or not.

[YOCTO #8231]

(Bitbake rev: 3ea10f4c16a557e94781251f6776b13acb8e9eba)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:59:59 +01:00
Elliot Smith ef6fc2bc1a bitbake: toaster: Replace "Run again" button with help text for cli builds
It's not possible to run a command-line build again, as Toaster
doesn't have access to the data used for that build.

Replace the "Run again" button with an icon which pops up some
help text to that effect.

Add test to check that the run again button is hidden and the
help icon displayed instead for command-line builds.

[YOCTO #8231]

(Bitbake rev: b67ac9e7cbab50951847dd1a63b12f41bb345dbb)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:59:59 +01:00
Elliot Smith 7467b6802d bitbake: toaster: Exclude "command line builds" project from projects typeahead
The "New build" drop-down provides autocomplete for Toaster projects.

However, it should not include the default project (for command
line builds), as it should not be possible for the user to
select this project as the container for a Toaster build.

[YOCTO #8231]

(Bitbake rev: d6210cd75b0cd8b6d24d5e99f607ba5cc45daf97)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:16:17 +01:00
Elliot Smith b5624c77ed bitbake: toaster: Show 'not applicable' for default project machine and release
The machine and release for the default project should show as
'not applicable' on the all projects page, as that information
isn't available for command-line builds.

Modify the templates with some conditionals to check for the
default project row, plus some data-* attributes to mark
where that data is to make testing possible.

Add some tests for the all projects page to ensure that
the correct machine/release are still shown for non-default projects,
and 'not applicable' for the default project.

[YOCTO #8231]

(Bitbake rev: ce27b3fd728f0373aa1adc0d47baace264529b45)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:16:17 +01:00
Elliot Smith 3c4c984e33 bitbake: toaster: Reorganise and lint tests
Get test suite ready to accommodate new tests for how command line
builds are shown on various pages.

(Bitbake rev: 1ee1fc5dcdbb26c9f6e04b7719d7196083212d4c)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:13:24 +01:00
Logan Buchy 3ba43f26ee bitbake: fetch2/hg: Include missing errno import
errno.ENOENT checked if deletion of the fullmirror fails.
Exception was thrown since module not imported

(Bitbake rev: d92ebfc34b69ad5df2d151e6b8299fbb5afa3e5f)

Signed-off-by: Logan Buchy <logan.buchy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:13:24 +01:00
Ed Bartosh 6fa3fecb12 bitbake: cooker: normalize build targets
BuildStarted event not fully represents build tasks for
the targets. If -c option is used to specify default task
it's not included into the event.

Made build targets to always look as <target>:do_<task>.
Consider default task (do_build or specified by -c command
line option) when normalizing.

(Bitbake rev: 0b0e214e6f53c97ad3d48f622c7fc0ca149956f6)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:13:23 +01:00
Michael Wood 5effe8f632 bitbake: toaster: Allow any text input to machine configuration variable
Allow any text input to the machine variable; as we may not have discovered
all the available machines until after a build.

[YOCTO #8418]

(Bitbake rev: f44b34833f164daf34c57703429ed8f122888037)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:13:23 +01:00
Ed Bartosh 320d05ea9f bitbake: toaster: exit or return depending on the mode
When toaster script is sourced 'return' should be used as
using 'exit' will cause script to exit the shell and close terminal
window. If script is called 'exit' should be used. Otherwise shell
will throw and error:
  return: can only `return' from a function or sourced script

Used 'return' or 'exit' depending on the toaster mode using
TOASTER_MANAGED variable.

(Bitbake rev: 643c42069a2788e9f40fa1bc9845f32aed3d096c)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:13:23 +01:00
Ed Bartosh 2e2e40c2c4 bitbake: toaster: set TOASTER_MANAGED variable
Set TOASTER_MANAGED to 1 if toaster is run as a script.

(Bitbake rev: 9f68f317e33d617558fce3427a94e14ef65c7d88)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:13:23 +01:00
Ed Bartosh a73895e492 bitbake: toaster: get rid of SRCFILE
Used TOASTER variable instead of SRCFILE as they're
essentially the same.

(Bitbake rev: 4b8cb1098ad73b4e9dc65b5c6de7d11d2e2468c7)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:13:23 +01:00
Ed Bartosh 779539cfb9 bitbake: toaster: use path to the script to guess config path
Used $TOASTER variable(path to the toaster script) instead of
current directory name to get path to the toasterconf.json

This way script should work in both modes and it doesn't depend
on the current directory.

(Bitbake rev: 5368d9af4e2c121938fc2421a536fb7f20a987fd)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:13:23 +01:00
Elliot Smith eb8b2b9ec8 bitbake: toaster: Guard against builds with no targets
Although this probably can't happen in practice (builds without
targets), this constraint is not currently enforced in the data model.

In the unlikely event that a build has no target (e.g. in test cases),
this causes a template rendering error.

Rather than rework the data model, add a guard to the template
to prevent it from rendering the target name if there are no
targets associated with the build.

(Bitbake rev: 4f409d8c01bae898ea142bd1417db99e12067753)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:13:23 +01:00
Elliot Smith 65e8bde9f1 bitbake: toaster: Remove Toaster exceptions section of build dashboard
Fatal build errors were displayed as exceptions, and highlighted
with less severity than they deserved.

Roll back to treating Toaster exceptions as errors by removing
the toaster_exceptions member on Build objects and displaying
EXCEPTION events in the errors section on the dashboard.

[YOCTO #8320]

(Bitbake rev: 59197320bdcefddf06084e871f1b5b21b21cbb63)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:13:23 +01:00
Elliot Smith 93f0b61749 bitbake: toaster: Record critical errors
Critical errors (where a build failed for reasons of
misconfiguration, such as a machine being specified which is not
in a project's layers) were being ignored (only log records
up to ERROR level were being logged to Toaster's db). This meant that
the build would fail but would not correctly report why.

Add support for CRITICAL error levels to the LogMessage model,
include errors at this level in the errors property for a build,
and show errors at this level in the build dashboard.

[YOCTO #8320]

(Bitbake rev: b6eacbca9cacb607de864ab7d093deb296da8226)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:13:22 +01:00
Elliot Smith 069a611097 bitbake: toaster: Test that exception isn't thrown by project page
Add a test which checks that an exception is no longer thrown
for the /toastergui/project/X page for the default project.

Note that we still get a spinning dialogue box on this page
because the default project has no configuration to display,
but at least it doesn't fail altogether.

[YOCTO #8277]

(Bitbake rev: 8795667d03bd8705d7e13c5d3d6bb6da371fa91d)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:13:22 +01:00
Elliot Smith 026e981264 bitbake: toaster: Check whether buildrequest exists before using it
Builds initiated from the command line don't have a buildrequest
associated with them. The build.buildrequest association is
only added if a build is triggered from toaster.

Some of the code for displaying the status of a build refers
to build.buildrequest without checking whether it has been set,
which causes an error to be thrown.

Add a guard to check whether the buildrequest has been set.

[YOCTO #8277]

(Bitbake rev: af33409612139ab2347baf6b847b23faea19752d)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:13:22 +01:00
Elliot Smith 1feeb8e499 bitbake: toaster: Always run bldcontrol migrations
The toaster startup script conditionally migrates the database
tables depending on whether you are in managed mode or not. This
means that if you are in analysis mode, some of the bldcontrol*
database tables used by managed mode are not available.

As a consequence, some of the code in toaster which refers to
those tables can break in analysis mode, as there's no clean
isolation of the two modes.

To prevent this from happening, always run the migrations for
managed mode and create the bldcontrol* tables, even if in
analysis mode.

Also clean up the function which starts up toaster so the
logic is easier to follow.

[YOCTO #8277]

(Bitbake rev: b1fc592131286ebbede2693be8c86636f0039011)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:13:22 +01:00
Michael Wood ae82d778e6 bitbake: toaster: buildinfohelper Detect command line builds
When we're building using toaster as just a listener to bitbake
(analysis mode) we need to handle the case where the toaster configuration data
isn't present so we don't need to try and update the existing information.

(Bitbake rev: a22faae2c3a5948356ce3cbc73c34509de65d370)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:13:22 +01:00
Belen Barros Pena 596c219bf4 bitbake: toaster: Disable add layer button when input is empty
The 'add layer' button in the project configuration page
is enabled when you select a layer from the type ahead.
However, if you delete the layer name, the 'add layer'
button remains enabled, and if you click it, the last
selected layer from the type ahead will be added to the
project.

It is probably better to disable the 'add layer' button
when the input field is empty.

[YOCTO #8449]

(Bitbake rev: d4820f16c76398400ddd573db03c654c51d40c1b)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:13:22 +01:00
Belen Barros Pena 24e5a178e1 bitbake: toaster: Have 'Version' next to recipe name
In all Toaster tables we show the version next to the recipe
or package names, with one exception: the recipes table in
the layer details page, where we are showing the version
next to the recipe description.

This patch moves the version column next to the recipe name,
for consistency with all other Toaster tables.

(Bitbake rev: b5af7084b28158a8a9eaf78db463555ae8e0f620)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:13:22 +01:00
Belen Barros Pena c895838189 bitbake: toaster: Improve directory structure layout
File names in the directory structure often wrap
due to big indenting for nested files and directories,
unnecessary width in certain columns, and not using
colspan when the symlink and package cells are empty.
The wrapping makes the table harder to read.

This patch reduces the amount of indenting, limits the
width of the 'Size', 'Permissions', 'Owner' and 'Group'
columns, and sets colspan to use the white space of the
symlink and package names when empty.

(Bitbake rev: 6167ee5a7569d8f841c340e672645cc133ea5a31)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:13:21 +01:00
Ed Bartosh 2f52ef4b2f bitbake: toaster: importlayer Update property names for importlayer api calls
Update the property names used in checking for existing layers. The
server side API changed but not all references were updated.

[YOCTO #8300]

(Bitbake rev: 1cbf0cf77a638257e18066f911fe4c4e13c278c6)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:13:21 +01:00
Michael Wood d302c98822 bitbake: toaster: Fix missing tooltips from layers on project configuration page
Re-enable the layer tooltips on the project configuration page.
This adds the required fields to the API used for the layer dependencies
to be able to show the metadata needed.

Also fixes link hrefs which were missing in the add layers confirmation
dialog (bug 8251).

[YOCTO #8295]
[YOCTO #8251]

(Bitbake rev: c7cb8255d0ab1fd7715e878c1c83ef0cd920387e)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-13 15:35:49 +03:00
Ed Bartosh 7e5464b451 bitbake: toaster: Fix broken test case
Fixed test_show_tasks_in_projectbuilds broken by latest changes in
project build view.

(Bitbake rev: 3c72c7634ab69a5eb18aa20a5c6d16a3e2666f62)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-12 14:38:57 +01:00
Ed Bartosh 2e375e63bf bitbake: toaster: exclude recipes with empty names
Modified get_all_compatible_recipes function to exclude recipes
with empty names from the result queryset. This should stop UI
to show recipes with empty names in compatible recipes page.

[YOCTO #7969]

(Bitbake rev: f7c1bd49e2e28d12c6604f5ae54bc96e1e7b6946)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-12 14:38:57 +01:00
Ed Bartosh fa3e82d952 bitbake: toaster: delete recipe if it can't be saved
Due to Recipe model constraints some recipes can't be
saved. However, they still can create incomplete records in
the database. This causes all sorts of errors when Toaster
operates with those objects. Removing them should fix those
issues.

[YOCTO #7969]

(Bitbake rev: 4d76a9e418fd98a7882aa29f974a7389f9689314)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-12 14:38:56 +01:00
Belen Barros Pena 82675fc4ed bitbake: toaster: Remove project name from latest project builds
The 'latest builds' list in the project builds page does not
need to show the project name, since it lists only builds
for the selected project.

This patch removes the redundant project name.

(Bitbake rev: 065652b9c649135f9e2fc5d9ba90e98f560dccdd)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-12 14:38:56 +01:00
Ed Bartosh 6aeaca1c83 bitbake: toaster: test get_alldeps API
Added test case to test Layer_Version.get_alldeps API.

(Bitbake rev: 159aa333c2f6344b5b37911c3f09601b9f0df6d8)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-12 14:38:56 +01:00
Ed Bartosh 0fb6be01fe bitbake: toaster: fix orm tests
Cleaned up and fixed orm tests. Removed test_build_layerversion as
it's not needed due to changed compatible_layer_versions API.

(Bitbake rev: 73ea29ed065bfaa80ee368b2a38c157e36fe1676)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-12 14:38:55 +01:00
Ed Bartosh dea679af1a bitbake: toaster: fix NameError
Fixed exception: NameError: global name 'DoesNotExist' is not defined

(Bitbake rev: eee5311b867d3c8c33e06d04e103bfd3647146f4)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-12 14:38:55 +01:00
Ed Bartosh 6e0c0fd835 bitbake: toaster: use get_alldeps in layerdetails renderer
Used Layer_Version.get_alldeps api in layerdetails template
renderer to get list of layer dependencies.

[YOCTO 8004]

(Bitbake rev: 077d0f41a3bd9cf2802a1488f1d6156ccac7df1d)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-12 14:38:54 +01:00
Ed Bartosh bd2ec77510 bitbake: toaster: implement API to get full list of deps
Implemented Layer_Version.get_alldeps API to recursively get
full list of dependencies for the layer. Dependencies that are
already in the project are filtered out from the result.
Result list of Layer_Version objects is sorted by layer name
for UI to look consistent.

This API is going to be used to show amount and list of
dependencies for the layer in the list of compatible layers
for the project.

(Bitbake rev: 7d853a3054a9ae3d18eb6f5bc13ba27d2795c31a)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-12 14:38:54 +01:00
Michael Wood e35c404537 bitbake: toaster: Don't descend into directories for cached_layers
As toaster makes use of clones of itself we have a mechanism to just
clone the current checkout and use that as a basis for the git clone for
the layers inside. For instance, in poky, if we're running in that directory,
we don't ever want to change the current checkout so we make a copy of
ourselves to do this work in.

This steps through cached_layers and previously-traversed directories
inside the current checkout to find any pre-existing checkouts.

This was ending up traversing too many directories down, so remove
the traverse and only deal with the current directory.

[YOCTO #8463]

(Bitbake rev: 9c40b3b6377ab8f5d6ac9b8e00585b71de00bf74)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 08:12:58 +01:00
Michael Wood 2d6701f190 bitbake: toaster: Update JS unit tests
Update JS unit tests.
- Expand the add remove layer check to make sure that the layer is
  actually added to the project.
- Remove some unused vars
- Make sure that the layers/project ids will always exist at the point
  of running the test.
- Add the missing typeahead input fields to the dom to fix the failing
  typeahead test.

(Bitbake rev: 46af40b95f842aa14ef7e3f0d516aef3899d5e42)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 08:11:30 +01:00
Michael Wood ab896df63b bitbake: toaster: Fix stale layer state buttons
Move the "is available to the project" state computation to the template
for the Layer add/remove buttons, Recipe build/Add layer as done for the
Package add/remove. This is more reliable as we can get an inconsistent
state on the front end JS as there are many opportunities for hitting
out of date project information.

[YOCTO #8294]

(Bitbake rev: 43469c3360566ad4897785f14f8717a9bc8b6078)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 08:11:29 +01:00
Michael Wood 41a5f82b4a bitbake: toaster: tables Add the recipe filter back into the Recipe table
Add the "Is recipe currently available in the project" filter back to
the Recipe table which was removed when we had intermediate
AvailableRecipe tables.

(Bitbake rev: b3682d1d851e616efa0715f9d43815a92e259432)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 08:11:29 +01:00
Michael Wood 2bebcd4995 bitbake: toaster: Fix typo in returning pk list of layer versions in current project
This was accidentally returning the primary key of the ProjectLayer
rather than the ProjectLayer.Layer_Version.pk

(Bitbake rev: b20f3626148e89af0e9fcfca911a5a1e4e355a41)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 08:11:29 +01:00
Michael Wood d6d680d2e5 bitbake: toaster: layerdetails update build recipe button class name
Update the class name which we are using for the recipe build button.
This fixes a regression in the buttons being enabled/disabled when
adding and removing the layer being viewed from the project.

(Bitbake rev: cc63f10f69105205e65b5f9647232b2b4b23ad48)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 08:11:28 +01:00
Elliot Smith 7794b5728d bitbake: toaster: Hide "Download build log" button if log doesn't exist
Our builds pages show all builds, but also include build requests
which may have resulted in a build failure, before the build
started (e.g. at the recipe parsing stage).

In such cases, the BuildStarted event is not captured by Toaster,
so we have no idea where the log file for the failed build is.

The result is that a build is shown by the Toaster UI /builds/ pages,
but it is really a pretend build which never went beyond being a
build request, and which has no associated log file. In turn, this
breaks the "Download build log" button on the build dashboard,
as there's no log file associated with the build.

Fix this by hiding the "Download build log" button for builds
which don't have a cooker_log_path.

[YOCTO #8373]

(Bitbake rev: 89e6cd03aa11c886f28520557af6c7ad51827b0e)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 08:11:28 +01:00
Ed Bartosh 8c695393cd bitbake: toaster: fix naming for clone directory
Toaster uses git url and branch to make a clone directory
name. Current code leaves '@' and '%' characters unchanged,
which can cause generation of wrong directory names.

Fixed this issue by replacing '@' and '%' with underscore.

(Bitbake rev: e076888a2120a37c388930073694750735d86507)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 05:39:50 +01:00
Michael Wood 41286f4730 bitbake: toaster: buildinfohelper Skip packages we have no build info about
If there are more packages listed as installed than we know about from
bitbake, and therefore have insufficient information to be able to
create a Toaster Package object then skip it. Also handle the case where
a dependency references such a package.

Also clarify the error logging.

(Bitbake rev: b4ce793685f70cab3f28cb4329aaaf3878cd62e8)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 05:39:04 +01:00
Michael Wood 97d000677e bitbake: toaster: buildinfohelper associate build data with built_recipe
Make sure we associate build data with the built recipe rather than
toaster's configuration copy of the recipe.

(Bitbake rev: 34d4ef7289d72d151ad0acdccab8b99c8c31221e)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 05:39:03 +01:00
brian avery 0dcc96323c bitbake: toaster: remove bashisms so script works in dash as well
[YOCTO #8217]

(Bitbake rev: be2d04c1831608922f90da03cb26d7fd5c863a47)

Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 05:33:06 +01:00
brian avery 8068aa3b0f bitbake: toaster: get rid of interactivity in bldcontrol
removed layer dir questions and base off TOASTER_DIR

removed build dir questions and base off TOASTER_DIR

base configuration file off of TOASTER_CONF

fixed some pylint issues

[YOCTO #8217]

(Bitbake rev: d5811968b5f22093365f381fb7e75ab46e5269c2)

Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 05:33:06 +01:00
brian avery 7d7823e597 bitbake: toaster: check for configuration file and exit if not found
[YOCTO #8217]

(Bitbake rev: 49e30c1aa8d717adbdcc15c90a668000789a6961)

Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 05:33:05 +01:00
brian avery 315989cd95 bitbake: toaster: remove layer and build dir interactive questions
added TOASTER_CONF env var for toaasterconf.json

added TOASTER_DIR env var for working dir

added bugfix so WEB_PORT env variable is honored

(Bitbake rev: c4b351e40060a359d59ab51b5ed2f9713ca1a9a5)

Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 05:33:05 +01:00
brian avery 489d5ff1c9 bitbake: toaster: removed superuser question from startup
added comment explaining how to create one later

(Bitbake rev: 782b7c74d5f10ed255538ba0c975ff342a34f5bf)

Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 05:33:04 +01:00
Michael Wood c7d1dab1c4 bitbake: toaster: orm Machines filter don't pass self in as parameter
Fix typo don't pass self in as a parameter, this evaluated to true
giving the wrong results meaning the machines typeahead did not return
valid results.

(Bitbake rev: 55ba889ef8900c95447861fa3985ca9cfe06afdf)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 05:33:04 +01:00
Elliot Smith dd957fe0f2 bitbake: toaster: Rationalise mimetype guessing to fix artifact downloads
Artifact download links were broken because the function to
get the mimetype for the artifact was incorrectly using the
underlying mimetype library. The function was also attached
to the build environment controller, which was unnecessary, as
we only support local controllers anyway.

Remove the mimetype getter on the build environment and
use the one in the view code instead. This works correctly
and prevents the download error from occurring.

[YOCTO #8369]

(Bitbake rev: 805fb2a9388c728600596e9b845a5c7eeaebd99c)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 05:33:04 +01:00
Elliot Smith ce9011a9de bitbake: toaster: Use Python's mimetypes module
filemagic is used to guess the mimetype of files when a user
requests a download. However, this adds a dependency on an
external library.

Python does have a mimetypes module, though this guesses the
mimetype rather than doing anything clever with the actual
file content. But for our purposes, it's more than adequate.
(NB Django also uses this module when serving static files.)

Use this instead of relying on any external code, and remove
the filemagic dependency.

(Bitbake rev: 0dd0ac25d54c73f13812db04826b57b3d16ea43f)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 05:33:03 +01:00
David Reyna 466bbec242 bitbake: toaster: display warnings for bad "IMAGE_FSTYPES" values
Display warning message for IMAGE_FSTYPES when no value is selected or
when the filter does not have any matches

[YOCTO #8126]

(Bitbake rev: 9a825eb928cb35096d2c1563788310fb6a13e93e)

Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 05:33:03 +01:00
Belen Barros Pena 8b7d846c68 bitbake: toaster: Set default columns in recipes tables
The compatible image recipes and software recipes tables
show pretty much all their columns by default. That's a few
too many columns. The default columns in both tables should
be: recipe, version, description, layer and build.

This patch sets the above as the default columns. It also
changes the table heading 'Recipe Version' to just 'Version',
which is shorter and self-explanatory.

[YOCTO #8421]

(Bitbake rev: 1f7bfe5e13bc39bb7eb6e039fe4b6291fc95d531)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 05:33:03 +01:00
Belen Barros Pena 9daf6ef908 bitbake: toaster: Comment out broken sorting and filters
YOCTO #8131 records that the sorting by 'errors', 'warnings'
and 'time', and the filtering by 'errors' and 'warnings',
are broken in the 'all builds' and 'project builds' pages.

To avoid exposing broken functionality to users, comment
out the sorting and filtering in those columns until we
we have a fix for YOCTO #8131.

(Bitbake rev: 1f15557efc922bf460640eeaf1622453419bc9a6)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 05:33:02 +01:00
Elliot Smith b661f535e6 bitbake: toaster: Don't HTTP cache ToasterTable responses
Django allows generated pages to be cached by default by the
browser. This can result in stale data displaying for some pages.

Instead, disable HTTP caching of ToasterTable responses, so that
each time a ToasterTable view is displayed, its data is
refreshed. This carries a performance penalty, but ensures that
ToasterTable views (e.g. compatible layers) are correctly
refreshed if the user navigates their history with forward/back.

[YOCTO #7660]

(Bitbake rev: 44dccd3018554915868d6c8fe5e22624a2fcdec5)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 05:33:02 +01:00
Elliot Smith a3742a0f68 bitbake: toaster: Don't add new history entries when table data loads
When ToasterTable data is loaded into the UI, a new entry is
added to the browser history. This means that pressing the back
button appears to have no effect, as you end up at the same page,
possibly with slightly different data.

Instead, use replaceState(), so that the browser history doesn't
grow, but the page context still gets updated.

[YOCTO #7660]

(Bitbake rev: 70c5e40a0f77ae4835fb95275621c345f8190240)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 05:33:02 +01:00
Ed Bartosh fa68ae0455 bitbake: toaster: use meaningful logging levels
Changed logging levels to more appropriate ones.

(Bitbake rev: 27d0360d13af0c698bf3a224b3f0d415f17bb678)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 05:33:01 +01:00
Ed Bartosh bd8b27b6a9 bitbake: toaster: ignore ReachableStamps event
Added ReachableStamps event to the list of known events to
ignore. This should stop toaster throwing error message:
ERROR: Unknown event: <bb.event.ReachableStamps>

(Bitbake rev: cd4137e13af6964858640b78aa7fe6f1612be251)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-11 05:33:01 +01:00
Richard Purdie 8fb5a5a338 bitbake: bitbake/lib: Update version to 1.28.0
(Bitbake rev: 3e63abc6977bd5ef52ccb4d3757536bd564dfd34)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-01 15:17:53 +01:00
Richard Purdie cdc57f6116 bitbake: siggen: Make it clear why nostamp tasks signatures don't match
If you run bitbake-diffsigs against two differing sigdata files from
nostamp tasks it shows no difference despite the differing checksum.

Change the code so this shows up as a nostamp 'taint' and at least
makes the issue clearer to the end user.

(Bitbake rev: 97679d18955dadaa34f9450564e44da99984d140)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-01 07:43:38 +01:00
Richard Purdie 1630f0a151 bitbake: runqueue: Add handling of virtual/xxx provider mappings
This firstly prints debug messages which show how bitbake decided to resolve
the virtual/xxx providers which is useful for debugging.

If the siggen has a tasks_resolved() method, it calls this, passing in
the mappings, allowing that to do things with the resolved names.

(Bitbake rev: d473fc84acddfd69a7207affcd89f65ea2ecf730)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-01 07:43:38 +01:00
Richard Purdie 0b96e6f1b5 bitbake: taskdata: Add a function to return the virtual/ mapping data
When building an execution task graph, bitbake does resolve virtual/xxx
namespaces into specific providers. This data isn't exported anywhere
however.

This adds a function so that runqueue can at least retrieve this data
which can then be used by the system.

(Bitbake rev: ce51a51482d0900060512b24503714a730d72266)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-01 07:43:38 +01:00
Richard Purdie 40fae3260c bitbake: cookerdata: Rename BBPKGS -> BBTARGETS
BBPKGS is a confusing name from before we tried to straighten out our
terminology. Its also a mostly unknown variable that isn't in wide use.

I've been asked about it recently and before people start relying more
heavily on it, I'd like to rename it BBTARGETS which better describes
what it does. Its not currently in the manuals, I'd prefer to document
it under the better name. I've not provided any migration path for the
variable since I believe its unused currently.

It allows the targets to built to be specified from a conf file in
addition to those on the commandline.

(Bitbake rev: f60c6a2172bceeb5682dcb738a02c4bf26176566)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-01 07:43:38 +01:00
Richard Purdie 1e467b333f bitbake: bitbake-worker: Guard against multiprocessing corruption of event data
In the forked child, we may use multiprocessing. There is only one event
pipe to the worker controlling process and if we're unlucky, multiple
processes can write to it at once corrupting the data by intermixing it.

We don't see this often but when we do, its quite puzzling. I suspect it
only happens in tasks which use multiprocessng (do_rootfs, do_package)
and is much more likely to happen when we have long messages, usually many
times PAGE_SIZE since PAGE_SIZE writes are atomic. This makes it much more
likely within do_roofs, when for example a subprocess lists the contents
of a rootfs.

To fix this, we give each child a Lock() object and use this to serialise
writes to the controlling worker.

(Bitbake rev: 3cb55bdf06148943960e438291f9a562857340a3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-01 07:43:38 +01:00
Ed Bartosh 4bc3f0994e bitbake: toaster: move clones into subdirectory
By default Toaster clones layers to the same level where
poky clone is. This can look messy if a lot of layers are
used for Toaster builds.

Moving them into _toaster_clones/ subdirectory should make
directory structure looking cleaner. It also safer to isolate
toaster clones from what user might create.

(Bitbake rev: 139e851831eea682aba0f9403dcc2eea7c1b05bd)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:39 +01:00
Ed Bartosh 9e1516d2bf bitbake: toaster: make clone directory name unique
Changed naming scheme for clone directory. Used full git url and branch to
make it unique. This should fix the issue with using the same git
repository, but different protocols, e.g. git://some.git.repo and
http://some.git.repo.

[YOCTO #8101]

(Bitbake rev: a9baf856a1ef7d84a47fa6615353f5b3fdb8a73b)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:39 +01:00
Ed Bartosh 552fd83d82 bitbake: toaster: fix reimporting module
removed second 'import re' from localhostbecontroller.py

(Bitbake rev: 30a9271ffa2834d9a4ffafe5c03ef9e874460419)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:39 +01:00
Ed Bartosh 55dc9279c8 bitbake: toaster: fix bug in resetting git repository
git reset --hard should be given either commit id or
origin/<ref name> to work properly. Without this fix git will
complain that origin/<commit id> does not exist.

[YOCTO #7505]

(Bitbake rev: ec05beff7d1b06e4df98199925c7102f5684f4e0)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:39 +01:00
Ed Bartosh 693934031a bitbake: toaster: use git reset --hard instead of rebase
Replaced 'git checkout <ref> && git rebase' with 'git reset --hard' as
with git checkout repository ends up with detached HEAD. Rebase makes
things even worse as it can cause conflicts. git reset --hard resets
repository to the required state in a most straightforward and
reliable way.

[YOCTO #7505]

(Bitbake rev: f3641a8422e67890410594453b8baf362c27df2b)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:39 +01:00
Ed Bartosh 3d73dfa223 bitbake: toaster: don't use --single-branch when cloning
git clone --single-branch works only with ref names. It fails if
commit sha1 is set in layer configuration with this error:
fatal: Remote branch <commit sha1> not found in upstream origin

Cloning repository without using --single-branch should work for
refs and commit sha1.

[YOCTO #7505]

(Bitbake rev: edac7b7533d50455baaaf05a46b7ace49bb24b95)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:38 +01:00
Markus Lehtonen 226e7da26e bitbake: utils: only add layer once in edit_bblayers_conf()
Prevent edit_bblayers_conf() from adding layer(s) multiple times. This
happened when BBLAYERS variable was "listed" multiple times in
bblayer.conf - i.e. the configuration was split into multiple separate
assignments.

[YOCTO #8316]

(Bitbake rev: 5e423237f9f4ff7e7e03bf066b0142ba4bd82219)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:38 +01:00
David Reyna d48b7ef607 bitbake: toaster: display most recent builds for projects
Display the most recent builds in the given project's build page.

[YOCTO #8186]

(Bitbake rev: ce9a7f1819a1897878ce154b3ee7e727a76165b0)

Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:38 +01:00
Michael Wood f902dc646d bitbake: toaster: orm remove the complicated querying on the ORM
We no longer need to compute each layer_version and all the recipes
which belong to this.

[YOCTO #8147]

(Bitbake rev: 505979ab931e3a2a218d7030d6064987e8f9ff14)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:38 +01:00
Michael Wood fe29297c6a bitbake: Revert "bitbake: toaster: don't re-create Target objects"
This delete followed up the foreign keys and deleted things that were
not expected to be deleted.

This reverts commit 08000eb27e.

(Bitbake rev: 46b119eb62a5a612fe4c0847862d34f408e556f7)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:38 +01:00
Michael Wood e6d967dba2 bitbake: toaster: buildinfohelper Create a copy of the built layer and recipe
Create a copy of the built layer and the recipes associated with it.
This is so that the user can view the historical information about a
build. i.e. a snapshot of the layer version and artifacts produced at
that build.

(Bitbake rev: 0683d9a2b15e3234a94437aaebac84bfcca1420b)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:38 +01:00
Michael Wood 17fe16bbf4 bitbake: toaster: tables show all recipes in the layerdetails even duplicates
For some layers we have multiple recipes for the same software with
differing versions. Change to showing all versions and add a version
column to the table instead.

(Bitbake rev: 8480e275e87275921d0dd54f2a7a445b1cd40203)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:38 +01:00
Michael Wood aed6d2e8c0 bitbake: toaster: Prioroitise the layer more generic vcs reference over the sha
When we do a build we update the last commit value that the layer was built at
However in future builds we do want to use the named reference rather
than the commit sha, e.g. master/fido

(Bitbake rev: a3d47eb923dd2cb5259f387c79549822807eca49)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:38 +01:00
Michael Wood 922503f4c1 bitbake: toaster: Create a relationship between build information and toaster layers
Previously this layer relationship was done by trying to match path
information that came back to the buildinfohelper with trying to query for
data in toaster's layers table. This rarely matched due to the lose coupling.

(Bitbake rev: 838e77c7c3c4006abd1327343a004590ab652de9)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:38 +01:00
Michael Wood 0bc0a44aff bitbake: toaster: Special case the openembedded-core layer to avoid duplicates
If the openembedded-core layer is specified in the toasterconf we need
to treat it differently because we may also get this layer either from
the layerindex source or I can also be provided locally.

(Bitbake rev: 8e36b6848c14f2708e1f87d12fb533ca0b596c65)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:37 +01:00
Ed Bartosh e68f63a1bd bitbake: toaster: Add test cases for new Image customisation features
- Adds tests for new ToasterTables
- Adds tests for new ReST API

co-author: Elliot Smith <elliot.smith@intel.com>
           Michael Wood <michael.g.wood@intel.com>

(Bitbake rev: 69f4ece18d70825d620f1d360749d587ea16f2a0)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:37 +01:00
Michael Wood d98c771a9a bitbake: toaster: Add Image customisation frontend feature
Add the Image customisation front end feature to Toaster.
Caveat - This feature is currently in development and should not be
enabled by default.

(Bitbake rev: 543586462b66434741f47f2884b4ccdeda5397b5)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:37 +01:00
Michael Wood 37948cc5d0 bitbake: toaster: Add ToasterTables for Image customisation feature
- Split up the recipes tables into Image recipes and Software Recipes
- Add CustomImageRecipe table
- Add SelectPackagesTable table
- Add NewCustomImagesTable table

(Bitbake rev: 4e5472e9ba6850081baa9d56fabc4ddb1aa24846)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:37 +01:00
Ed Bartosh a3ff4b28ba bitbake: toaster: Add new ReST API for Image Customisation feature
Implemented xhr_customrecipe API. To create a custom recipe from a
base recipe.
Implemented xhr_customrecipe_packages API to add/remove packages
to/from custom recipe.

co-authored see Signed-off-by

(Bitbake rev: 84be400237173970716616eeab6a37d776aa011b)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:37 +01:00
Michael Wood 28153acb0a bitbake: toaster: Fix indentation of jsunittests view
Fix indentation to 4 spaces

(Bitbake rev: 076945ea026091dc709f7cfea01ef119d0572bf3)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:37 +01:00
Ed Bartosh 60f3ddb2fb bitbake: toaster: implement decorator for REST responses
Implemented xhr_response decorator to decorate responses
from REST methods into Django HttpResponse objects.

This decorator should reduce amount of repeated code in
REST methods and make them more readable.

(Bitbake rev: bb0696f343aca44207581f15ff2b4f0045f7530c)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:37 +01:00
Michael Wood a7f43bd532 bitbake: toaster: add toggle for enabling image customisation feeature
This feature is currently under heavy development and should be used
with caution.

(Bitbake rev: ffc11b2c6c6bac4643233cc46418b025c94607c8)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:37 +01:00
Michael Wood 3ff6401723 bitbake: toaster: Add CustomImageRecipe model
This model lists custom image recipes for the project.
It is populated when new custom image is created.
It holds reference to the base recipe and list of packages
included into custom image.

For CustomImageRecipes the packages will be copied in and
therefore not associated with a build so Remove the requirement
for the package to have a Build.

co-author: Ed Bartosh <ed.bartosh@linux.intel.com>

(Bitbake rev: 33cbf4cd3b4ca47c4901501f5f1eafdfdfdae023)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:37 +01:00
Michael Wood 8948d04bfd bitbake: toaster: ToasterTable remove unused class definition
Remove ToasterTemplateView as this isn't used by anything.

(Bitbake rev: aa0b2dc6789cb6ae12511a4a930a4118337a162e)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:36 +01:00
Michael Wood c1157cff9f bitbake: toaster: add nocache option to the ToasterTable widget
Useful for skipping the cache mechanism when debugging.
Simply append nocache=true to the end of the url with the table in
it.
Also adds a debug message if the table is using cached data.

(Bitbake rev: 3492c1570db9ecd728d9901bda1f02a2d7495f19)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:36 +01:00
Michael Wood 1cafc39e96 bitbake: toaster: widgets ToasterTable Add more info to search field exception
If we get a search fields exception then also print out the model name

(Bitbake rev: 8152db1e35b9a3f35a25f801008f6fc61a6b11c4)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:36 +01:00
Michael Wood c71bbad38a bitbake: toaster: widgets ToasterTable add logger to notify when cache hit
Useful for debugging to remind you if you're hitting a cache or not

(Bitbake rev: 68b9c2d95ab17ee2d16325862342126eb5c3c1a7)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:36 +01:00
Ed Bartosh 934f8d7b23 bitbake: toaster: create custom layer and recipes for Image customisation
When building customised recipes toaster creates custom layer directory
and puts layer.conf and custom recipes to it.

[YOCTO #8075]

(Bitbake rev: f81b48c30a548bee946d34c56aa1872785bcec30)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:36 +01:00
Michael Wood 340b3983db bitbake: toaster: tables Move the title and name into the widget
For historical reasons this was being set in the urls definition. We
can set this in the actual definition of the table and defaults in the
widget.

(Bitbake rev: 2f7949caec3c9194328c35b2e075eaae85409efe)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:36 +01:00
Ed Bartosh e1851fe04c bitbake: toaster: make a workaround for old style index
For a new style indexes 'inherits' field is used to determine if
recipe is an image recipe.

As old style indexes don't have 'inherits' field this can be
guessed from recipe name. Let's consider recipe an image recipe
if recipe name contains '-image-'.

(Bitbake rev: d769d3f54faaf1a3091b1eb6ed1d17a8a108210a)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-29 14:11:36 +01:00
Leonardo Sandoval f78f90240a bitbake: prserv/serv.py: Better messaging when starting/stopping the server with port=0
When starting the server using port=0, the server actually starts with a
different port, so print a message with this new value. When stopping the
server with port=0, advise the user which ports the server is listening to,
so next time it tries to close it, user can pick up the correct one.

[YOCTO #8560]

(Bitbake rev: 851e53a216682fc9133f51c05a24527cfc677741)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-28 12:00:33 +01:00
Leonardo Sandoval 134b267ebb bitbake: prserv/serv: Close the DB connection out of class destructor
When launching the PR server daemon, the PRData __del__ function was being
called (no reason found yet) where the DB connection closed, thus following
PR updates were not getting into the DB. This patch closes the connection
explicitly, not relaying on the __del__ function execution.
Closing the connection in turn causes all WAL file transactions to be moved
into the database (checkpoint), thus effectively updating the database.

[YOCTO #8215]

(Bitbake rev: c1b4754f69003df1a83fafc1c80a9ef74400b6dd)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-28 12:00:32 +01:00
Leonardo Sandoval 5a51fb28db bitbake: prserv/serv: Start/Stop daemon using ip instead of host
In cases where hostname is given instead of an IP (i.e. localhost
instead of 127.0.0.1) when stopping the server with bitbake-prserv --stop,
the server shows a misleading message indicating that the daemon was not
found, where it is actually stopped. This patch converts host to IP values
before starting/stopping the daemon, so it will always work on IP, not on
hostnames, avoiding problems like the latter.

[YOCTO #8258]

(Bitbake rev: bd6398e967c234e89d773f509512ebf460fa76ff)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-24 17:55:05 +01:00
Cristian Iorga 9f66aa1a11 bitbake: toaster: start script warning text formatting small improvement
Add a carriage return to build environment warning message to
avoid it being  mixed up with shell prompt.

(Bitbake rev: d3675cca419946cc19b3f280446fe1f656f11902)

Signed-off-by: Cristian Iorga <cristian.iorga@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-23 22:44:56 +01:00
Markus Lehtonen c6eaef01c5 bitbake: tinfoil: remove logging handler at shutdown
Otherwise the logger gets multiple handers (and the user get duplicate
logging output) if another tinfoil instance is initialized after one is
shut down().

(Bitbake rev: 74d67be7a4b591fab2278f7c184f282d11620c62)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-23 22:44:56 +01:00
Belen Barros Pena fb26ea3605 bitbake: toaster: remove time from builds in progress
In the 'latest builds' section of the all builds page,
completed builds show the time they completed at. Builds
in progress should not display such time, since they
haven't completed yet.

This patch removes the time information that was showing
for builds in progress, whatever that time actually was.

(Bitbake rev: 0df02c87573bf9a66df2e424cd9534e67a8ab3dc)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-23 22:44:56 +01:00
Elliot Smith 15b482b16b bitbake: toaster: Add fake entry to Target_File for filesystem root
The files-in-image.txt file is produced by bitbake after an
image is created, listing all the files in the image.
However, this list doesn't include the root directory ('/').

buildinfohelper.py then tries to construct the filesystem
tree from this file, assuming that every directory apart from
the root directory (which is special-cased) can be assigned
a parent. But because the root directory isn't listed in
files-in-image.txt, an object for the root directory is never
created.

The direct subdirectories of the root ('./bin', './usr' etc.)
then can't be assigned a parent directory, as the object
representing the root directory doesn't exist. This
results in a Target_File lookup error and causes the
directory listing page to fail.

Fix this by creating a fake entry for the root directory
in the Target_File table, so that the direct subdirectories
of / can be assigned a parent. Note that it doesn't matter
that the root is faked, as its properties are never shown
in the directory structure tree.

[YOCTO #8280]

(Bitbake rev: a4015768183e5a3fa39a6c2b4dea0088ca182d80)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-23 22:44:55 +01:00
Michael Wood 767fe69fd0 bitbake: toaster: layerdetails Fix back button tab behaviour
This completes the behaviour fix of the back button in the layerdetails
page as we not only have parameters in our history we also have the hash
to indicate which tab is active. As we pop our history we need to show
the corresponding tab.

[YOCTO #8252]

(Bitbake rev: 8fc7f94af19cd8489a944b02d9a406bd62d001fa)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-23 22:44:55 +01:00
Ke Zou 4c0320ffc0 bitbake: toaster: UI test improvements
* Add decorator in logging mechanism
* Add more debug information

(Bitbake rev: ab94f4bbef38d23e7e8be0663781eaecf84f0172)

Signed-off-by: Ke Zou <ke.zou@windriver.com>
Signed-off-by: Stanciu Mihail <stanciux.mihail@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-23 22:44:54 +01:00
brian avery 4c5af777ea bitbake: toaster: support selenium testing from mac OS X
* added a mac section to the cfg file
* added mac specific screenshot code

(Bitbake rev: bf8748aafc2291bb814fe0ec8a28d5eed9a1d5f0)

Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-23 22:44:54 +01:00
Ed Bartosh e6c497097f bitbake: toaster: add 2 UI tests
Tested that UI shows task names for the builds in
both all-builds and projectbuilds views.

(Bitbake rev: 092b1a9eebbd3f0747f6152c63182f18bccb2054)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-23 22:44:54 +01:00
Ed Bartosh f6a70adcfb bitbake: toaster: change UI to show tasks
Changed toaster UI to show tasks if they're specified for the
builds and use them when restarting builds.

[YOCTO #7442]

(Bitbake rev: 3c196c15f0ae4c6ac2b92e0a75562962f3da0089)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-23 22:44:53 +01:00
Ed Bartosh 08000eb27e bitbake: toaster: don't re-create Target objects
Due to re-creating Target objects from bitbake events task information
stored in original objects is lost.

There is no valid reason to remove existing objects. It's safer to query
them instead of re-creating as original object contain more information
than events coming from bitbake.

(Bitbake rev: aab4aff75eefb31aa53885d7735feee5daa294aa)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-23 22:44:53 +01:00
Ed Bartosh ea373586d3 bitbake: toaster: store task name in Target objects
Information about a task is not stored in Target objects.
This makes it impossible to correctly operate with the builds
where task is specified.

Storing taks name is an enabler for other fixes in UI and backend
related to restarting builds.

(Bitbake rev: 0a69a8a18075c976ed8681d9d75529f8c2f48514)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-23 22:44:52 +01:00
Romain Perier 7b86c771c8 bitbake: bitbake: bb.fetch2.git: Import errno module
Currently this module is dereferencing errno.ENOENT but the python module "errno"
is not imported, which causes a crash when fetching from a git repository.

(Bitbake rev: 93e4c9bb2393b1074f5a01e7eaaac742a59d8086)

Signed-off-by: Romain Perier <romain.perier@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-19 22:38:44 +01:00
Alexandru DAMIAN e993aa6705 bitbake: toaster: hide irrelevant builds in the project builds view
This patch fixes the project builds view so it doesn't show
"in progress" builds or builds for other projects.

Note that this also modifies the "all builds" view to use
the same queryset filtering as the project builds. This is
to avoid excluding "in progress" builds more than once, which
is what was happening before.

The patch also has a minor change to ensure that when
displaying the project builds page, only builds for that
project are in the results.

The queryset filtering is now split over several lines so
you can see what's going on.

[YOCTO #8236]
[YOCTO #8187]

(Bitbake rev: 771c08b9be1f7875e0216e381ab0a81ef0d26256)

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-19 11:31:59 +01:00
Richard Purdie 5b4e380b9b bitbake: data_smart: Ensure OVERRIDES dependencies account for contains()
The dependencies of OVERRIDES were not including DEFAULTTUNE in OE-Core.
This is pulled in by a bb.utils.contains() reference which the override
dependency tracking code wasn't accounting for.

This patch ensures we do track contains references too.

(Bitbake rev: f3ee534cb0560dbb5f88a0ffe01e9305bae102e1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-18 15:14:04 +01:00
Richard Purdie 60d019dde0 bitbake: data_smart: Correctly handle OVERRIDE values set using ??=
Right now, OVERRIDES dependency variables set using ??=, e.g. TARGET_ARCH
in OE-Core don't have their dependencies tracked. This is a bug, fix it.

(Bitbake rev: 944734503768f9e9223ef041f2d7873455418a54)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-18 15:14:03 +01:00
Richard Purdie 701ad76270 bitbake: data_smart: When considering OVERRIDE dependencies, do so recursively
Sadly its not enough to consider the dependencies of OVERRIDES, we
need to resolve their dependencies and so on recursively. If we don't
do this, some variable can be changed and the resulting data store is
incorrect.

(Bitbake rev: 82143ac064d391300e762ba7520ef1f8df18b574)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-18 15:14:03 +01:00
Richard Purdie 4325f6f03d bitbake: data_smart: Expand overrides cache recursively
If the values that make up OVERRIDES are themselves overridden,
we end up into some horrible circular logic. Unfortunately some
metadata does depend on this functionality.

e.g:
DEFAULTTUNE_virtclass-multilib-xxx = Y
which changes TUNE_ARCH
which changes TARGET_ARCH
which changes OVERRIDES

As a solution, we iterate override expansion until the values don't
change. If we iterate more than 5 times we abort and tell the user to
report the issue.

(Bitbake rev: 10279697c701e01bf6fdd5e9f92792ef5134807b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-18 15:14:02 +01:00
Christopher Larson 3a5e46b6ed bitbake: bb.fetch2.{git, hg}: remove tarball if it needs updating
We were maintaining state in the form of ud.repochanged to determine whether
we need to write the tarball in the case where the tarball already exists, but
this state is maintained only in memory. If we need to update the git repo,
set ud.repochanged to True, and then are interrupted or killed, the tarball
will then be out of date.

Rather than maintaining this state, simply remove the out of date tarball when
we update the git repo, and it will be re-created with updated content in
build_mirror_data.

[YOCTO #6366]

(Bitbake rev: eaaa81393f181432c8586b17ade623f42c9fed2e)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-18 09:05:36 +01:00
Elliot Smith 441f04ce97 bitbake: toaster: Simplify redirects when build page parameters are missing
A RedirectException is used to redirect the client to the
correct page when their original request is missing the
required parameters (page, orderby etc.). However, the code
is difficult to follow.

Rather than catching RedirectExceptions and rethrowing them with
different view URLs, ensure that the RedirectException has the
correct URL in it when thrown by passing the original page
name to the _build_list_helper() method.

Modified from an original patch by
David Reyna <david.reyna@windriver.com>.

(Bitbake rev: 38f935647dd768a912b933adebfc9cb225a01a54)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-18 09:05:36 +01:00
Elliot Smith 30f9f79fb9 bitbake: toaster: Fix date range pickers on the project builds page
These were referring to the old HTML elements with ids
"date_from_created" and "date_from_updated", but their
ids have changed to "date_from_started_on" and
"date_from_completed_on". This meant that they weren't
functional.

This fixes the references.

Modified from an original patch by David Reyna
<david.reyna@windriver.com>.

(Bitbake rev: fe9a554249b623ef515502043fdbf50e1ac62a3a)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-18 09:05:35 +01:00
Elliot Smith 047245fff4 bitbake: toaster: Show correct builds count on project pages
The counter for completed builds on the project pages
includes builds in progress.

Instead use the completedbuilds queryset to count the
number of completed builds for display on project pages.

Modify how the completedbuilds queryset is constructed so
it only excludes builds with status "in progress".

(Bitbake rev: 455a0087e0dcd74998abd02a110942f25da127be)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-18 09:05:35 +01:00
Alexandru DAMIAN 5528f3a8fd bitbake: toaster: hide irrelevant builds in the project builds view
This patch fixes the project builds view so it doesn't show
"in progress" builds or builds for other projects.

Note that this also modifies the "all builds" view to use
the same queryset filtering as the project builds. This is
to avoid excluding "in progress" builds more than once, which
is what was happening before.

The patch also has a minor change to ensure that when
displaying the project builds page, only builds for that
project are in the results.

The queryset filtering is now split over several lines so
you can see what's going on.

[YOCTO #8236]
[YOCTO #8187]

(Bitbake rev: 09079f15c0511a6d17ce1cc29be6de5387e45f09)

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-18 09:05:35 +01:00
Mark Hatle 2bb600a579 bitbake: tests/fetch.py: Fix recursion failure in url mapping
Instead of reproducessing the same line over and over and over, we remove the
current line from the mirror list.  This permits us to re-evaluate the list
while excluding all matches that have previousily occured.

Without this fix, adding this test results in a failure:

RuntimeError: maximum recursion depth exceeded in cmp

(Bitbake rev: 24a8e9a5b0ba145ae589178d74365c986ebca325)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-18 09:05:34 +01:00
Mark Hatle 4c3d4eceb1 bitbake: fetch2/__init__.py: uri_replace regex handling
We should only substitute one time.  If we do it without a max count, we can
end up matching over and over.

Before this change:

https://downloads.yoctoproject.org/releases/bitbake/bitbake-1.0.tar.gz

with a mirror of

https://.*/[^/]*    http://AAAA/A/A/A/

would end up either recursing indefinitely or result in:

http://AAAA/A/A/A/A/A/A/A/A/A/bitbake-1.0.tar.gz

(Bitbake rev: 4d254e02e2867dd9a6663508c8ca9f2733af71a8)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-18 09:05:34 +01:00
Elliot Smith 460e4c2c2c bitbake: toaster: Don't def a function for each call to build_artifact()
Cache the mimetype object and only define the function for
getting a mimetype once.

Also ensure that filemagic is listed as a requirement of
toaster. Doing this also means we can remove the code
which tries multiple different "magic" libraries, as we know
we have the right version available.

(Bitbake rev: 8d3aa2d46ebab7a59e57234f0b3f6fc3225a13e8)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-18 09:05:33 +01:00
Elliot Smith b6d1d2acd5 bitbake: toaster: Avoid unnecessary local to local copy of cooker log
The cooker log is copied from its original (bitbake) location
to an artifact directory chosen by the user (chosen when
checksettings.py runs as part of the managed mode;
it's one of the annoying questions you're asked at startup).
The copy happens as part of the runbuilds script run, which
is started in a loop from the toaster startup script
in managed mode.

When a user requests the log for a build via toaster, they
are getting the log which has been copied to the artifact
directory, not the original bitbake log.

This works for the managed case, where the runbuilds command is
running in a loop and copying log files for completed builds to
the artifact directory. However, in analysis mode, there are two
problems:

1. checksettings isn't run, so the artifacts directory isn't
set. toaster is then unable to figure out where the log files
should have been copied to.

2. The log files aren't copied to the artifacts directory
anyway, as runbuilds isn't running in analysis mode.

To fix this, just point the user to the local bitbake log file
in its original location. This avoids the copy step, and means
we can remove a whole question from the toaster startup sequence,
as we no longer need an artifact directory.

The only downside to this is that it's not appropriate for
remote bitbake servers. We will need to revisit this and
possibly reinstate the copy step once we have to reconcile
local and remote builds and make their logs available in
the same way.

[YOCTO #8209]

(Bitbake rev: 5697bbcc88edad85891f66d28b8803a9c9d27ff2)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-18 09:05:33 +01:00
Elliot Smith 8c63d6049d bitbake: toaster: Read correct cooker log path from toasterui
The BB_CONSOLELOG variable changes by the time we read it in
BuildInfoHelper. This means that the log file location we
are using is incorrect, so the links to the cooker logs don't
work.

Instead, read it at the point when the BuildStarted event occurs
in toasterui. The BB_CONSOLELOG variable has the correct value
here, so pass that to BuildInfoHelper.

[YOCTO #8209]

(Bitbake rev: 20609eebee0d2318806cf81913e7ce6dc1005507)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-18 09:05:33 +01:00
brian avery 013c030c42 bitbake: toaster: delete multiple builds cleanup
This patch cleans up the multiple delete. It:

1) skips build id's that don't exist rather than giving a traceback.
2) let you pass in the ids as a space separated list
3) fixes the usage to match the space separated list format

[YOCTO #7726]

(Bitbake rev: a065f7e5e9c07dbd71a98e7db1d7f711607716f3)

Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-18 09:05:32 +01:00
Richard Purdie 3165af39bf bitbake: data_smart: Separate out update_overridevars into separate function
No code changes, just refactoring to allow for functionality
changes by moving things to a separate function.

(Bitbake rev: 2eb934814179ccf42e3d424dabe26b17d013a7ed)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-18 09:05:32 +01:00
Richard Purdie 07aef86c3a bitbake: cache: Handle spaces and colons in directory names for file-checksums
If there is a space in a directory name containing a file in file-checksums
(e.g. from a file:// url), you currently get tracebacks from bitbake. This
improves the code to handle colons and spaces in the file-checksums names
since it possible to figure out the correct names.

[YOCTO #8267]

(Bitbake rev: 87282b283921a58426f24fb21151db457c5bca66)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-18 09:05:31 +01:00
Richard Purdie 68aefbb27d bitbake: bitbake-worker: Ensure pipe closure doesn't crash before killpg()
If the pipe is closed, we want to ensure that we kill any child processes
by triggering the sigterm handler before we exit. This code does that,
hopefully avoiding the remaining process left behind issues on the autobuilder.

(Bitbake rev: 60f6c2818f38c4d9c2d9aaa42acf3071636f4a3b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-12 22:50:14 +01:00
Ed Bartosh aaa48b69f3 bitbake: toaster: increase waiting time
Increased waiting time for toasterUI from 10 to 25 seconds. Bitbake
takes longer time to start, so toaster should wait longer.

[YOCTO #8240]

(Bitbake rev: 224ee3685fe20915b21d299ab80f7b289a916dca)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-12 22:50:14 +01:00
Belen Barros Pena 446070074e bitbake: toaster: Changes to navigation
This patch:

* Changes the breadcrumb to provide access to either
the project builds or the project configuration, as
appropriate

* Changes the left navigation in the project configuration
to reflect the hierarchical relationship between the
basic configuration and all other configuration pages

* Changes the left navigation in the build history to bring
it in line with the changes in the project configuration

This way the breadcrumb explicitly exposes the hierarchy
of the application, which is its correct behaviour, making it
easier to move around within Toaster.

(Bitbake rev: 135dff67216759286f584e501583584a9cb09f27)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-09 14:27:57 +01:00
Richard Purdie 6271fcb589 bitbake: bitbake-worker: Handle SIGKILL of parent gracefully
If we SIGKILL cooker (the parent process), ensure the worker notices
and shuts down gracefully. To do this:

* trigger the sigterm handler if the parent exits
* ensure broken pipe writes don't trigger backtraces which interfer with
  other exit work
* notice if our command pipe is broken due to EOF and sigterm if so

(Bitbake rev: c43d6a8d711db8d3bd9a1976b9f8e3efdb4cb4ae)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-09 14:27:57 +01:00
Richard Purdie 3195cd92eb bitbake: bitbake-worker: Simple code cleanup
start/end are unused here and we can improve the code conditional blocks.

(Bitbake rev: 68f53dd77fe0bbfa044bd037a9484e0e1c9088b4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-09 14:27:56 +01:00
Richard Purdie 85256c8fa9 bitbake: prserv: SIGTERM and deamonization fixes
Worryingly, if you SIGKILL the bitbake cooker, an autostarted PR server
will remain behind. It turns out there are a few things we should do:

* The PR service doesn't need to daemonize when started from cooker,
  it just complicated the process lifecycle. Add a fork() method
  to handle this and use the non-daemon mode for the singleton.

* Reset the sigterm and sigint handlers. Bitbake cooker installs its
  own which we inherit meaning PR server was ignoring SIGTERM. Installing
  our own handlers which include a sync makes most sense here. Since
  we're in the code, make it sync the database on SIGINT.

* Use the new bb.utils.signal_on_parent_exit() call so that we get a
  SIGTERM when the parent (usually cooker) exits and we can shutdown
  too. Alternatives would be having an open pipe or polling
  os.getppid() for changes but this seems more effective.

(Bitbake rev: 05d31fa1f56bd3d3d363a16a421d9ba7541d4293)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-09 14:27:56 +01:00
Richard Purdie 289007e5eb bitbake: utils: Add signal_on_parent_exit() function
Add a new bb.utils.signal_on_parent_exit() function so that a process
can register to recieve a signal when the parent dies. There is no
POSIX standard for this and the implementation is Linux specific.
Alternatives would be having an open pipe or polling os.getppid()
for changes but this seems more effective and less invasive to most
of bitbake's code structure.

We need to be able to determine when parents die to ensure child
processes stop running in a variety of circumstances to avoid
locks being held and ensure clean shutdown.

Roughly based on https://gist.github.com/evansd/2346614

(Bitbake rev: 34974f5e30e9b09c016481e4c81c156a5f379784)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-09 14:27:56 +01:00
Richard Purdie 384533bdb4 bitbake: server/process: Handle SIGTERM more gracefully
Currently if you send a SIGTERM to the bitbake UI process, the system basically
hangs if tasks are executing. This is because the server process doesn't
actually try any kind of shutdown before exiting.

This patch trys executing a stateForceShutdown command first, which is
enough to stop any active tasks before the system exits.

I also noticed that terminate can execute multiple times, once at SIGTERM
from the handler and once from the real exit. Double execution leads to
stack traces and potential hangs (writes to dead pipes), so ensure
the code only can run once.

With these fixes, bitbake much more correctly deals with SIGTERM to the
UI process.

(Bitbake rev: 1032ddddbe3241da02ebb3608a1c40f9123b9e80)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-09 14:27:55 +01:00
Belen Barros Pena db92bc2388 bitbake: toaster: replace ETA with % of tasks done
The ETA we show for builds in progress is woefully
inaccurate. In the 1.8 release we replaced it with
the % of tasks completed. Somehow, we regressed to
the ETA, so bringing the task % back.

(Bitbake rev: a841dc85770ea5c6fa8cf06ba5fdfe214e69afb2)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-08 23:54:15 +01:00
Belen Barros Pena c386abc89e bitbake: toaster: fix project names in latest builds
Make sure that the project name we show for each build
in the latest builds section of the all builds page:

* Has the same styles as the build, depending on the
build status (in progress, failed or success)

* Links to the project page

(Bitbake rev: 09abcf3199b2e86758a974a47ebe31f5fb79440a)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-08 23:54:15 +01:00
Belen Barros Pena a4dbd4bfa1 bitbake: toaster: remove unnecessary white space
There was too much white space above the first
heading in the all builds page. This patch removes
the class that was adding the extra space.

(Bitbake rev: c9b20009c5e161987a143b00ee22218a4c9c0692)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-08 23:54:15 +01:00
Richard Purdie 1e9ac4c5cc bitbake: toastergui: Add missing files from 1c2f6b9b7b9e700146944b9d6d2114e0d014ee81
(Bitbake rev: cf2b6b621cb483aa1a1a66c09fc431efc207c91e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-08 23:54:14 +01:00
Christopher Larson a0246316c6 bitbake: bb.fetch: don't remove the clone when an update fails
When our clone exists, but is out of date, and the attempt to update it fails,
we don't necessarily want to remove the entire clone, particularly if it's
a large repository.

(Bitbake rev: 19af272ba5256653edeff6acbceeb09e3e478d61)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-04 16:24:00 +01:00
Christopher Larson ba50c2f4b9 bitbake: bb.fetch: handle checksums consistently for mirrors
If the main fetch method doesn't support checksums, the user will not be
defining them in the recipe, so we don't want to check them for
premirrors/mirrors either. This ensures that we never error due to missing
checksums on a git mirror tarball.

(Bitbake rev: 24c79bbed361b37f12d3351af13602e3d4386f4c)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-04 16:24:00 +01:00
Christopher Larson abba11daf9 bitbake: bb.fetch.git: remove leading '.' from gitsrcname
When using an absolute file URI, there's no host, and the path starts with
'/', the dir under ${DL_DIR}/git2/ ends up starting with '.', so is hidden.
Remove any leading '.' to fix this.

(Bitbake rev: 8dce6964d56b36a77fb113f2ad496cc992a5ff36)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-04 16:23:59 +01:00
Christopher Larson 7c4fdb8c17 bitbake: bb.cookerdata: include useful traceback for ExpansionError/ParseError
Show the user only the portion of the traceback which was from the metadata,
nothing from bitbake's internal calls.

(Bitbake rev: c45054aef03393fa0bf70e853ddcfc55988493cf)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-04 16:23:59 +01:00
Christopher Larson 5fe590cd6e bitbake: bb.data_smart: retain traceback from ExpansionError
This gives us the needed context of the original ExpansionError, which is
invaluable when we have a chain of function calls in the expansion.

(Bitbake rev: c514b6fbea77ede1b7871b89592a33ed39b1d71c)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-04 16:23:58 +01:00
Alex Franco 7ef2f951d6 bitbake: cooker/runqueue: Allow bitbake commands starting with do_
The output of "bitbake, -c listtasks pkg" lists tasks with their real names
(starting with "do_"), but then "bitbake -c do_task" fails, as "do_" always
gets unconditionally prepended to task names. This patch handles this error
by checking whether a task starts with "do_" prior to prepending it with it
when the task runlist is being constructed (and a few other corner cases).

[YOCTO #7818]

(Bitbake rev: dd3050ceef37ac556546e940aa596ce96ef6c8df)

Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-04 16:23:58 +01:00
Belen Barros Pena ccc1d9ac0c bitbake: toaster: improve explanation of checkout dir
Improve the explanation we show to users during set up
when asking them to set the layers checkout directory.

The patch also makes minor changes in text formatting to
improve legibility.

[YOCTO #7740]

(Bitbake rev: dd9284944ae0f0feecb70adab880fed636f7cd59)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-04 16:23:57 +01:00
Richard Purdie 4b9de767f0 bitbake: event/server: Add _uiready flag to handle missing error messages
If you start and suspend a bitbake execution so the bitbake lock is held,
then try and run "bitbake -w '' X", you will see bitbake return an error exit
code but print no message about what happened at all.

The reason is that the -w option creates a "UI" which swallows the messages. The
code which handles this exit failure mode thinks a UI has printed the messages
and therefore doesn't do so.

This adds in an extra parameter to the UI registration code so that we
can figure out whether its a primary UI or not and base decisions on whether
to display information on that instead. This fixes the error shown above and
some bizarre failures on the Yocto Project Autobuilder.

[YOCTO #8239]

(Bitbake rev: d1d60a68c2de40c2984d5040d14251c1be121b0b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-04 16:23:56 +01:00
Benjamin Esquivel 7c079463c9 bitbake: utils: Fix a potential error in movefile
bitbake utils' movefile is now prone to malform the destination
file with duplicated file name strings. Fixing it to force a file
name append iff the dest argument is a dir not a file name

(Bitbake rev: 38dd27f7191da002a16c561be3790ce487045b01)

Signed-off-by: Benjamin Esquivel <benjamin.esquivel@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-04 16:23:56 +01:00
Elliot Smith bc6a1a23e3 bitbake: toaster: Add tests for pages which show the default project
Test that the correct landing page redirect is applied, depending
on the state of the default project.

Test that the default project is only shown on the /projects page
if it has at least one build.

[YOCTO #7932]

(Bitbake rev: 85a65d2b652c2ccc6cfb90fd8bc9048d0e72341e)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-03 16:52:50 +01:00
Elliot Smith 3b6d193cd7 bitbake: toaster: Fix test for projects page JSON
The tests expect data to be returned in a particular format,
which doesn't include is_default or the num_builds fields
added to support default project improvements.

Add the extra fields so the tests pass again.

[YOCTO #7932]

(Bitbake rev: 66f965b68dda76a755d88879191a0f340ba19142)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-03 16:52:50 +01:00
Elliot Smith 3b233fca49 bitbake: toaster: Exclude default project unless it has builds
Don't include the default "command line builds" project in
the projects view unless it has builds associated with it.

[YOCTO #7932]

(Bitbake rev: b37318057ccbfb3889d28fd593cc1602febf1ae9)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-03 16:52:49 +01:00
Elliot Smith c9a05be1ff bitbake: toaster: Only redirect to projects page if user has added projects
The landing page currently redirects the user if there are any
projects in the db. Because we now always have at least one
(the default one added by a migration), we always get the redirect.

Change this so that when the user hits the landing page,
we only redirect them to the projects page if there is at least
one user-added project and there are no builds.

[YOCTO #7932]

(Bitbake rev: 316a70da93055b63eff7d40b7fd8bd07299074af)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-03 16:52:49 +01:00
Elliot Smith 214ba674d0 bitbake: toaster: Improve how default project is identified and fetched
Command line builds are associated with a "default project"
(as we currently require a build to have a project). This
acts as a container for builds initiated outside Toaster.
Currently, this project is marked as the default by its ID
being 0. However, this doesn't work with MySQL, as MySQL
won't allow 0 in a foreign key which references an
autoincrement field.

Instead, use an is_default field to track the default Project
for builds initiated outside Toaster.

Add a method to fetch this default project, rather than fetching
a project with a magic ID.

Add this default project in a migration, rather than as a side
effect of a get_or_create() style method.

Also ensure that builds always have a project explicitly assigned
to avoid any magic with a build's project foreign key defaulting to
0 (as it no longer does).

[YOCTO #7932]

(Bitbake rev: 71b709a1bbc26d89d61873763b467d21e625b274)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-03 16:52:48 +01:00