Removed dependency to argparse and wsgiref as they're in
standard python library.
wsgiref crashes during installation with python 3 as it's not
compatible with python 3.
Added dependency to pytz as it's used in the toaster code.
(Bitbake rev: 46f185afaaef74c566eec6d8fc1edc68a3b96b32)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Replaced local imports with absolute imports.
Used .module for local imports.
This should make the code to work on python 2 and python 3.
[YOCTO #9584]
(Bitbake rev: 3f1e68c783308dcb51242d0fdeef758e581ccc8c)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some APIs have been moved to other modules in python 3:
getstatusoutput: moved from commands to subproces
urlopen: moved from urllib2 to urllib.request
urlparse: moved from urlparse to urllib.parse
Made the imports work for both python versions by
catching ImportError and importing APIs from different
modules.
[YOCTO #9584]
(Bitbake rev: 1abaa1c6a950b327e6468192dd910549643768bb)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Replaced compicated calls of reduce with more clear code.
As reduce was removed from python 3 this change is mandatory
for the code to work on both pythons.
Here is an example change for illustration purposes:
original code:
querydict = dict(zip(or_keys, or_values))
query = reduce(operator.or_, map(lambda x: __get_q_for_val(x, querydict[x]), [k for k in querydict])))
replaced with:
query = None
for key, val in zip(or_keys, or_values):
x = __get_q_for_val(k, val)
query = query | x if query else x
[YOCTO #9584]
(Bitbake rev: 249d0bc6094ec9f369a02b78d8ed634a239e5ee4)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Usage of force_bytes in BuildRequest.__str__ method caused
python 3 to throw "__str__ returned non-string (type bytes)"
error.
Replaced force_bytes with force_text to make the code working
on both python 2 and python 3.
[YOCTO #9584]
(Bitbake rev: 9dd9c1393a84d1110c647e84253af8e0bb6acc45)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The upgrade to python3 is the final nail in the coffin for image-writer
and the goggle UI. Neither seem used or recieve patches and are based
on old versions of GTK+ so drop them, and the remaining crumbs support
pieces.
(Bitbake rev: ee7df1ca00c76f755057c157c093294efb9078d8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Disable the problematic gtk usage for use with pygtkcompat. The following
commit removes these tools/UIs entirely but we may as well leave this
piece in the history in case anyone does want a starting point for reusing
them.
(Bitbake rev: c53c7418d392452450352ca2175667dbdbd92401)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
get_context was added to mutliprocessing as part of 3.4.0
(Bitbake rev: 710351610e3ca4a1b61abc67564f84907e9b2f1c)
Signed-off-by: Jeremy Puhlman <jpuhlman@mvista.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This seemingly convoluted syntax doesn't work in python3. Instead
use the chained exception handling syntax which appears to make more
sense here.
(Bitbake rev: b19a4c5166303b1fa680582adf63e6a5564bfb4c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Various misc changes to convert bitbake to python3 which don't warrant
separation into separate commits.
(Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Upgrade to 4.4.1 which has been run through 2to3 as per the maintainers
recommendation for v3 use.
(Bitbake rev: 2f4b98af93c971a8c466ffaf3c09cca0edb6e3ad)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Under python the type conversions can mean there are float values
used for triggering the parse progress events which then fails.
Add an explict int() conversion to ensure the parse events are
generated under python3.
(Bitbake rev: 138329c58e92744c56aae3ab70ceeef09613250c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Task.SSTATE_NA and Task.SSTATE_MISS are both valid conditions for the
condition that a Task.OUTCOME_COVERED and Task.OUTCOME_PREBUILT.
(Bitbake rev: acac9bf660ac5939beb64d7b8d55541dea1c7b4e)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a port of the oe self test to the django test framework from
oe-core meta/lib/oeqa/selftest/_toaster.py
(Bitbake rev: 94418b1b2f9466d35461acdb982fd6b130b2331c)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a helper class for running build tests. Subclass this and call the
build method to get setup for running tests on the resulting data from a
build.
(Bitbake rev: d1882371cae6b9a8296ea739e6e46ca6a6dc9da8)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When running certain tests we want a particular database specified.
When bitbake toaster ui is being tested pass it these test settings so
that it uses the same database as the unit tests running.
(Bitbake rev: c52e34cac4362ba0a3cb3ea1fcb639e1d802aa85)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move the execution sequence for a build out of the polling loop and into
it's own fuction. This means that we can call the function on it's own
if we just want to trigger one build rather than infinite polling.
This is something needed for the build tests.
(Bitbake rev: e8e059602c31394a4be5293e2ca0b8434fb4a8cf)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We don't need to pass the DATABASE_URL around and read it back if we
setup the django framework in the correct way.
We make the default sqlite database path a full path so that the
database isn't being assumed to be in CWD.
Also add some more useful comments on the database settings.
This is preparation work to migrate the build tests and be able to
trigger builds on differently configured databases.
(Bitbake rev: 973c740404ca6a09feea250d3433075995067fe0)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add basic tests to validate the value user types
in the text box for IMAGEFS_TYPES. Added a test
case to show the checkbox get automatically selected
when user types value available in the check list.
Added a test case to verify if the check box is enabled
then the text box should also get updated accordingly.
[YOCTO #7828]
(Bitbake rev: 62c74eb38f44d98b40427edf56e40785b076a938)
Signed-off-by: Sujith H <sujith.h@gmail.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a label to clarify the purpose of the first input field, replace <p>
with <label> for the second set of controls, and give some top margin to
the validation message.
(Bitbake rev: 99b1b52fe4f2b593d7835d2db64eb028ed42600a)
Signed-off-by: Belen Barros Pena <belen.barros.pena@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This functionality helps users to add custom
image fs types available other than the checkboxes
in the UI. User can add imagefs types in the text
box and use them in the build.
[YOCTO #7828]
(Bitbake rev: c14cbb026598be064f08f46bb513456c7a3089f5)
Signed-off-by: Sujith H <sujith.h@gmail.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use open() instead of file() and close files when finished with them.
(Bitbake rev: 033c5a16ff19781ed793c2d97d285884017a2a4e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously we only tracked the flags (minus excluded) of variables we depend
on, but not the flags we use explicitly.
(Bitbake rev: bdeb3dcd7c92e62a7c079e7b27048c4114f24a3a)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This variable is a regex-escaped version of LAYERDIR, for safer use in
BBFILE_PATTERN, so as to avoid issues with regex special characters in the
layer path.
[YOCTO #8402]
(Bitbake rev: 72900522778b6ff08b135bf8bb97dff3f1a20bd9)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It's useful to see tracebacks for ExpansionErrors, but only if we skip the
leading bitbake-internal elements, otherwise we see elements of the expansion
process.
As one example:
Before:
ERROR: ExpansionError during parsing /scratch/yocto-new/external-as-needed/poky/meta/recipes-core/glibc/glibc-locale_2.23.bb: Failure expanding variable PV[:=], expression was ${@get_external_libc_version(d)} which triggered exception AttributeError: 'module' object has no attribute 'external'
After:
ERROR: ExpansionError during parsing /scratch/yocto-new/external-as-needed/poky/meta/recipes-core/glibc/glibc-locale_2.23.bb
Traceback (most recent call last):
File "PV[:=]", line 1, in <module>
File "/scratch/yocto-new/external-as-needed/meta-sourcery/recipes-external/glibc/glibc-external-version.inc", line 3, in get_external_libc_version(d=<bb.data_smart.DataSmart
object at 0x7f05d2566950>):
sopattern = os.path.join(d.getVar('base_libdir', True), 'libc-*.so')
> found_paths = oe.external.find_sysroot_files([sopattern], d)
if found_paths:
ExpansionError: Failure expanding variable PV[:=], expression was ${@get_external_libc_version(d)} which triggered exception AttributeError: 'module' object has no attribute 'external'
(Bitbake rev: 7ff5b9eed82b7f4fd138fc6d746a0b79efbea98a)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This uses bb.utils.load_plugins, based on the plugin handling in recipetool
and devtool in oe-core.
(Bitbake rev: 5e542df9b966a99b5a5b8aa7cf6100174aff54b2)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to flush the footer removal, else it may not be outputted until
the buffer is flushed as part of StreamHandler and this would lead to
it removing the ERROR output just printed which is extremely confusing.
Also ensure the footer is cleared before printing a summary as in
some cases it wasn't being removed, also leading to user confusion.
(Bitbake rev: 0e030c4d074c41859608dab5f3ad26b05f56b306)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The signature data file comparison functions are meant to be able to
handle data files containing just the base hash data. This had regressed
in some places so add fixes to allow these comparisons to be made. The
runtime components in the data files are optional.
(Bitbake rev: 2a6659fd748e255a02c2f9d047829d6edfe65317)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
knotty captures two levels of keyboard interrupt: a single interrupt
or two interrupts in a row. These then trigger stateShutdown
and stateForceShutdown respectively.
toasterui doesn't have an equivalent way of capturing interrupts and
using them to shut down bitbake. Now that we are no longer using
knotty + XMLRPCServer for our command line builds (since switching to
per-project build directories), we see some odd side effects of this,
such as builds continuing after they have been interrupted on the
command line.
Bring toasterui in line with knotty (copy-paste most of the code
in knotty.py which deals with interrupts) so that a keyboard
interrupt actually shuts down the bitbake server (if not in
observe only mode).
Additionally use the cancel_cli_build() method to set the Build
status to CANCELLED in Toaster's db when we get keyboard interrupts.
This means that builds interrupted on the command line show as
cancelled (same as if they'd been cancelled from the Toaster UI),
as specified in the UI designs.
[YOCTO #8515]
(Bitbake rev: d39d2edca95900da433074ee95a192d7bfe7090d)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This will be used from toasterui to cancel the current command-line
build when a keyboard interrupt is captured.
[YOCTO #8515]
(Bitbake rev: 1486c770327b53bb5e04baa5f3ea26d8154aed63)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After clean ups remove api assumptions. Our table data often contains
html snippets to display certain things such as dependency pop overs or
simply links to other parts of the UI. Take these into account when
testing the values of the table data.
(Bitbake rev: 7f61a68f2caba538e302ca9c2727bb43a6c77c64)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- Remove the unused 'computation' field
- Remove the code to try to make the tables behave like an api
- Remove custom JSON encoder in favour of DjangoJSONEncoder
- Simplify get_data and add comments
- Add exception type instead of using generic Exception
- Clean up python style warnings
(Bitbake rev: 16d8198e8f6668c5fa5467ff4bda86c5d66a6cad)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Used except 'except (<exception1>, <exception2>):' syntax as it's
supported by python 2 and pythone 3.
Old syntax 'except <exception1>, <exception2>:' is not supported
by python 3.
[YOCTO #9584]
(Bitbake rev: d19e305ffa44a848b02ede63dc5de8d2640089e6)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Used print() function instead of print statement
to make toaster script to work with both python 2 and python 3
[YOCTO #9584]
(Bitbake rev: 09d37ee51219edcd0be6fd24c82fce392533b39b)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Dictionary method has_key is deprecated in python 2 and absent
in python 3.
Used '<key> in <dict>' statement to make the code working on
both python 2 and python 3.
[YOCTO #9584]
(Bitbake rev: 3d7ad7ba0d1a6f688ae885817c049f2a8ced11b5)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if the server dies, its possible that log messages are never
displayed which is particularly problematic if one of those messages
is the exception and backtrace the server died with.
Rather than having the event queue exit as soon as the server disappears,
we should pop events from the queue until its empty before exiting.
This patch tweaks that code so that even if the server is dead and we're
going to exit, we return any events left in the pipe. This makes
debugging certain failures much easier.
(Bitbake rev: 29f6ade68fb2b506a23a7eb3a00cdcffa291b362)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now the event is sent unconditionally we can drop this feature
as its no longer needed.
(Bitbake rev: 473deeb0fc6065693e1fcfcbb8b79753103db537)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The data included in the event is useful for implementing a pre-build
check that warns about unexpected components, for example because of
an incorrect configuration or changed dependencies.
Such a check can be done in a .bbclass that gets inherited
globally. But in contrast to a UI, such a class cannot request that
the event shall be emitted, and thus the event has to be emitted
whether there is a consumer or not.
This was done conditionally earlier out of concerns about the
performance impact. But now events are handled more efficiently, so
that concern no longer seems valid: in some simple testing (admittedly
on a fast build workstation), the two lines (generating the data and
emitting the event with it) only took about 0.05 seconds (measured
with timeit). That was for a build with roughly 500 recipes (from
pn-buildlist aka depgraph['pn']), triggered via the command line. That
was even with a consumer of the data active and doing some work, so it
should be even faster when there is no consumer.
(Bitbake rev: 5ddaf5b7ed1001d2dd3f67e7a6d704afa85479d2)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using MySQL, the project builds info delivered by MySQL
differs from that delivered by SQLite: the former returns text
values from the enumeration for Build outcomes, while the latter
returns the integer value. This causes the progress bar JS to
break, as it is expecting outcome strings.
Modify the recent_build() method to include an outcomeText property
for each Build object, then use this in the conditionals in the
progress bar JS.
[YOCTO #9498]
(Bitbake rev: 7ac374adf1cc70173ff6cc492bc078bba1cf500b)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the cooker fails to start, ensure a correct exception is displayed to the
user. After handling any queued events simply re-raise the original exception
else the output can be unclear.
(Bitbake rev: 9a4db1aa608c17d31bf5ea1cab5a99beb565dd83)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The string formatting wasn't correct and we should exit if we hit
errors here similar to the other exception handlers.
(Bitbake rev: b90a16408a5c45ce5312384f278e19d09f8dda4d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
python3 can't cope with the previous approach we were using to pass
exceptions through the RPC. Avoid this by creating a formatted exception
on the sender side.
(Bitbake rev: d7db75020ed727677afbad07a90fb3eac0bf2c45)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this, the dict can reorder causing sanity test failures.
(Bitbake rev: ca8c91acc9396385834b266d4e8b84d917e5e298)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This avoids a deprecation warning in python 3.
(Bitbake rev: bf1a92d0c002d73e8a34472dced1343dc4a4251a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh
Richard Purdie
Jeremy Puhlman
Michael Wood
Sujith H
Belen Barros Pena
Christopher Larson
Elliot Smith
Patrick Ohly