Commit Graph

3181 Commits

Author SHA1 Message Date
Richard Purdie b301251668 bitbake: progress: Ensure missing start event is fired
The init function of the parent class fires a progress event for 0
progress rather than a start event. UI code was assuming that progress
events should always have a start event first. This change ensures that
the start event is correctly generated.

This fixes crashes that were seen in knotty in some configurations.

(Bitbake rev: 9841651e050a3e9f395ab3c62545c51197734584)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:27 +01:00
Richard Purdie 1b5688e0f0 bitbake: knotty: Handle process indicators more gracefully
Mistakes can happen with the generation of the progress events, change
knotty to be more tolerant of this rather than crashing, reporting to the
user when something unexpected happens. I haven't debugged why multiple
finish events appear to be triggered.

(Bitbake rev: 7dd06b1016b36420a9c55a45ff29dd64ae1dbcda)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:27 +01:00
Paul Eggleton 1b930b41a5 bitbake: runqueue: report progress for "Preparing RunQueue" step
When "Preparing RunQueue" shows up you can expect to wait up to 30
seconds while it works - which is a bit long to leave the user waiting
without any kind of output. Since the work being carried out during this
time is divided into stages such that it's practical to determine
internally how it's progressing, replace the message with a progress
bar.

Actually what happens during this time is two major steps rather than
just one - the runqueue preparation itself, followed by the
initialisation prior to running setscene tasks. I elected to have the
progress bar cover both as one (there doesn't appear to be much point in
doing otherwise from a user perspective). I did however describe it as
"initialising tasks".

(Bitbake rev: 591e9741e108487ff437e77cb439ef2dbca42e03)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:27 +01:00
Paul Eggleton f3b62c1c2e bitbake: runqueue: add ability to enforce that tasks are setscened
Add the ability to enter a mode where only a specified whitelist of
tasks can be executed outright; everything else must be successfully
provided in the form of a setscene task (or covered by a setscene task).
Any setscene failure outside of the whitelist will cause the build to
fail immediately instead of running the real task, and any real tasks
that would execute outside of the whitelist cause an immediate build
failure when it comes to executing the runqueue as well.

The mode is enabled by setting BB_SETSCENE_ENFORCE="1", and the
whitelist is specified through BB_SETSCENE_ENFORCE_WHITELIST, consisting
of pn:taskname pairs. A single % character can be substituted for the pn
value to match any target explicitly specified on the bitbake command
line. Wildcards * and ? can also be used as per standard unix file name
matching for both pn and taskname.

Part of the implementation of [YOCTO #9367].

(Bitbake rev: 624722c067a7fdd0c0f5d8be611e1f6666ecc4a2)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:27 +01:00
Paul Eggleton 76feac37ce bitbake: knotty: add quiet output mode
Quiet output mode disables printing most messages (below warnings) to
the console; however these messages still go to the console log file.
This is primarily for cases where bitbake is being launched
interactively from some other process, but where full console output is
not needed.

Because of the need to keep logging all normal events to the console
log, this functionality was implemented within the knotty UI rather
than in bb.msg (where verbose mode is implemented). We don't currently
have a means of registering command line options from the UI end, thus
the option actually has to be registered in main.py regardless of the
UI, however I didn't feel like it was worth setting up such a mechanism
just for this option.

(Bitbake rev: db95cdef08e339dec7462bfde3ad7d75c1c60dd8)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:27 +01:00
Paul Eggleton 465f93968f bitbake: knotty: show task progress bar
In addition to the "currently running n tasks (x of y)" message, show a
progress bar for another view on how much of the build is left. We have
to take care to reset it when moving from the scenequeue to the
runqueue, and explicitly don't include an ETA since not all tasks take
equal time and thus it isn't possible to estimate the time remaining
with the information available.

(Bitbake rev: de682015a3fefeff36ddc4197641a700f3fb558d)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:27 +01:00
Paul Eggleton 8d56d596bb bitbake: knotty: add code to support showing progress for sstate object querying
Add support code on the BitBake side to allow sstate.bbclass in
OpenEmbedded to report progress when it is checking for availability of
artifacts from shared state mirrors.

Part of the implementation for [YOCTO #5853].

(Bitbake rev: 070ae856da0715dbaf4c560c837ea796ffc29f00)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:26 +01:00
Paul Eggleton 0e3281f68b bitbake: lib/bb/progress: add MultiStageProgressReporter
Add a class to help report progress in a task that consists of multiple
stages, some of which may have internal progress (do_rootfs within
OpenEmbedded is one example). Each stage is weighted to try to give
a reasonable representation of progress over time.

Part of the implementation for [YOCTO #5383].

(Bitbake rev: 751b75602872a89e8b1a7c03269bc0fdaa149c6f)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:26 +01:00
Paul Eggleton ac5e720575 bitbake: lib: implement basic task progress support
For long-running tasks where we have some output from the task that
gives us some idea of the progress of the task (such as a percentage
complete), provide the means to scrape the output for that progress
information and show it to the user in the default knotty terminal
output in the form of a progress bar. This is implemented using a new
TaskProgress event as well as some code we can insert to do output
scanning/filtering.

Any task can fire TaskProgress events; however, if you have a shell task
whose output you wish to scan for progress information, you just need to
set the "progress" varflag on the task. This can be set to:
 * "percent" to just look for a number followed by a % sign
 * "percent:<regex>" to specify your own regex matching a percentage
   value (must have a single group which matches the percentage number)
 * "outof:<regex>" to look for the specified regex matching x out of y
   items completed (must have two groups - first group needs to be x,
   second y).
We can potentially extend this in future but this should be a good
start.

Part of the implementation for [YOCTO #5383].

(Bitbake rev: 0d275fc5b6531957a6189069b04074065bb718a0)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:26 +01:00
Paul Eggleton 1cf6e14a6c bitbake: knotty: import latest python-progressbar
Since we're going to make some minor extensions to it, it makes sense to
bring in the latest version of python-progressbar. Its structure has
changed a little but the API hasn't; however we do need to ensure our
overridden _needs_update() function's signature in BBProgress() matches
properly.

(Bitbake rev: c3e51d71b36cbc9e9ed1b35fb93d0978e24bc98a)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:26 +01:00
Paul Eggleton 481048cd2a bitbake: knotty: provide a symlink to the latest console log
If you're looking to find the latest console log repeatedly it can be a bit
tedious - let's just create a symlink just as we do with other logs to
make it easy to find.

(Bitbake rev: e9f41c0507a6527bf2ed86506813d4d4a89f8ebf)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:26 +01:00
Ross Burton 312f1a5e74 bitbake: fetch2/wget: attempt checkstatus again if it fails
Some services such as SourceForge seem to struggle to keep up under load, with
the result that over half of the autobuilder checkuri runs fail with
sourceforge.net "connection timed out".

Attempt to mitigate this by re-attempting once the network operation on failure.

(Bitbake rev: 54b1961551511948e0cbd2ac39f19b39b9cee568)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:26 +01:00
Robert Yang c6d50b2729 bitbake: lib/bb/build.py: remove task flag in deltask()
Otherwise the function like d.getVarFlag(e, 'task', True) which is used by
do_listtasks will still get it, and list the deleted tasks.

(Bitbake rev: 779d73619daf59f76f5b0313e7fb5409f6e82553)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-23 14:26:33 +01:00
Ed Bartosh 95f6e7bd0f bitbake: cooker: clean up EvertWriter
Restructured EventWriter code to make it more readable:
 - got rid of init_file method as it's called only once
 - renamed exception variable e -> err
 - renamed event variable e -> evt
 - simplified main 'if' structure of send method

(Bitbake rev: 31977e7bb98f676197c6cee66f6ab4c12d4dcbde)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-20 17:23:57 +01:00
Ed Bartosh f6fc25d1df bitbake: cooker: replace EventLogWriteHandler with namedtuple
class EventLogWriteHandler is a simple wrapper class with only one
class member. Replacing it with namedtuple makes code less nested and more
readable.

(Bitbake rev: 7c5b6812d32d173df36e7f9fc1d877329e79f994)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-20 17:23:57 +01:00
Ed Bartosh a158388a51 bitbake: cooker: don't remove event file
There is no need to remove output file as it gets rewritten by
open(self.eventfile, 'w') anyway.

(Bitbake rev: 1fc9957837b7038dfb983217a3fcd880f143e3a4)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-20 17:23:57 +01:00
Ed Bartosh e28b36e1e8 bitbake: cooker: encode event objects to base64
pickle converts python objects into the binary form that can't be
decoded to text and therefore can't be converted to JSON format.

Attempt to convert event objects raises this error:
TypeError:
    b'\x80\x03cbb.runqueue\nrunQueueTaskSkipped\nq\x00)...
    is not JSON serializable

Encoded pickled event objects to base64 to be able to convert data
structure to JSON.

[YOCTO #9803]

(Bitbake rev: f18055237e6084f90f6221442e3ba021dcc59c50)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-20 17:23:57 +01:00
Ed Bartosh 3e7edc303c bitbake: cooker: move EventLogWriteHandler to the top module level
EventLogWriteHandler object was created and used in
BBCooker.initConfigurationData.
This causes creation of multiple EventLogWriteHandler objects
and results in duplicated entries in the output event file
as BBCooker.initConfigurationData is called multiple times.

Added eventlogfile parameter to EventLogWriteHandler to avoid using
global variable DEFAULT_EVENTFILE.
Moved EventLogWriteHandler to the module level.
Created EventLogWriteHandler object in BBCooker.__init__ to ensure that only
one handler object is created.

(Bitbake rev: d3ad8eee850ec2df54aa09fae44cc7e69c12f32a)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-20 17:23:57 +01:00
Jussi Kukkonen 90d7b09af2 bitbake: depexp.py: port to gtk+3
The code is still a bit icky (and should be refactored to not use
Gdk.threads_enter/leave) but it should work about as reliably as
it did with Gtk+2.

Based on earlier patches by Maxin and Joshua.

(Bitbake rev: 8eee64a64144e27b5b8c2aca88e138882c3deab7)

Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-20 17:23:56 +01:00
George McCollister f74ba25c0a bitbake: runqueue: Use tid instead of taskid in find_chains()
In 2c88afb6 find_chains()'s taskid argument was renamed to tid but
taskid is still used as key to explored_deps dictionary. Use tid instead
of taskid.

(Bitbake rev: 29a34ae8f5306d2779bcc761c52f1f9d13a0c0c5)

Signed-off-by: George McCollister <george.mccollister@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-16 22:42:53 +01:00
Andrew Bradford 26447a0d0b bitbake: fetch2/perforce: Rework to support SRCREV and P4CONFIG
In recipes which use the perforce fetcher, enable use of SRCREV to
specify any of: ${AUTOREV}, changelist number, p4date, or label.  This
is more in-line with how the other fetchers work for source control
systems.

Allow p4 to use the P4CONFIG env variable to define the server URL,
username, and password if not provided in a recipe.

This does change existing perforce fetcher usage by recipes and will
likely need those recipes which use the perforce fetcher to be updated.
No recipes in oe-core use the perforce fetcher.

References [YOCTO #6303]

(Bitbake rev: 6298696bb94a127cdec7964315f6891ba92cd026)

Signed-off-by: Andrew Bradford <andrew.bradford@kodakalaris.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-16 17:37:58 +01:00
Stephano Cetola 9f91785cc5 bitbake: fetch2: fix unpacking of deb packages
Python 3 changed the return value of check_output to binary rather than
a string. This fix decodes the binary before calling splitlines, which
requires a string.

(Bitbake rev: 1072beefe172423873a22a10c7171e10d0401e1e)

Signed-off-by: Stephano Cetola <stephano.cetola@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-16 17:37:58 +01:00
Ed Bartosh 772e9b4eab bitbake: bitbake: fix wrong usage of format_exc
First parameter of traceback.format_exc is a 'limit' - a number
of stracktraces to format.

Passing exception object to format_exc is incorrect, but it works in
Python 2 as this code from traceback module works:
    while tb is not None and (limit is None or n < limit):
Comparing integer counter n with the exception object in Python 2
always results in True. However, in Python 3 it throws exception:
    TypeError: unorderable types: int() < <Exception type>()

As format_exc is used in except block of handling another
exception this can cause hard to find and debug bugs.

(Bitbake rev: a9509949d7e2adba6e3cd89f97daa19a955855b5)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-16 11:51:26 +01:00
Richard Purdie ff6d458f9a bitbake: parse/ast, event: Ensure we reset registered handlers during parsing
When parsing, we should reset the event handlers we registered when
done. If we don't do this, parse order may change the build, depending
on what the parse handlers do to the metadata.

This issue showed up as a basehash change:

ERROR: Bitbake's cached basehash does not match the one we just generated (
/media/build1/poky/meta/recipes-core/meta/nativesdk-buildtools-perl-dummy.bb.do_unpack)!

This is due to the eventhandler in nativesdk.bbclass being run, despite
this .bb file not inheriting nativesdk.bbclass. The parse order was
different between the signature generation and the main multithreaded
parse.

Diffsigs showed:

bitbake-diffsigs 1.0-r2.do_unpack.sigbasedata.*
basehash changed from 887d1c25962156cae859c1542e69a8d7 to cb84fcfafe15fc92fb7ab8c6d97014ca
Variable PN value changed from 'nativesdk-buildtools-perl-dummy' to '${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}'

with PN being set by the event handler.

(Bitbake rev: 0219271d4130c1f4cf071c7577a4101c54c04921)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-15 08:35:07 +01:00
Richard Purdie af04a52bf7 bitbake: taskdata/runqueue: Rewrite without use of ID indirection
I'm not sure what possesed me when I wrote this code originally but its
indirection of everyting to use numeric IDs and position dependent lists
is horrific. Given the way python internals work, its completely and
utterly pointless from  performance perspective. It also makes the code
hard to understand and debug since any numeric ID has to be translated
into something human readable.

The hard part is that the IDs are infectous and spread from taskdata
into runqueue and even partly into cooker for the dependency graph
processing. The only real way to deal with this is to convert everything
to use a more sane data structure.

This patch:
* Uses "<fn>:<taskname>" as the ID for tasks rather than a number
* Changes to dict() based structures rather than position dependent lists
* Drops the build name, runtime name and filename ID indexes

On the most part there shouldn't be user visible changes. Sadly we did
leak datastructures to the setscene verify function which has to be
rewritten. To handle this, the variable name used to specifiy the version
changes from BB_SETSCENE_VERIFY_FUNCTION to BB_SETSCENE_VERIFY_FUNCTION2
allowing multiple versions of bitbake to work with suitably written
metadata. Anyone with custom schedulers may also need to change them.

I believe the benefits in code readability and easier debugging far
outweigh those issues though. It also means we have a saner codebase
to add multiconfig support on top of.

During development, I did have some of the original code coexisting with
the new data stores to allow comparision of the data and check it was
working correcty, particuarly for taskdata. I have also compared
task-depends.dot files before and after the change. There should be no
functionality changes in this patch, its purely a data structure change
and that is visible in the patch.

(Bitbake rev: 2c88afb60da54e58f555411a7bd7b006b0c29306)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-15 08:35:07 +01:00
Richard Purdie 29b4fb1796 bitbake: runqueue: Change buildable/running lists to sets
Using positions in lists for flags is an odd choice and makes the code
hard to maintain. Maintaining a list is slow since list searches are
slow (watch bitbake -n slow massively with it) but we can use a set()
instead.

This patch uses python sets to maintain the lists of tasks in each state
and this prepares for changing the task IDs from being integers.

(Bitbake rev: 8c1ed57f6ea475b714eca6673b48e8e5f5f0f9c3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-15 08:35:07 +01:00
Ed Bartosh f0740b038f bitbake: toaster: replace viewkeys() -> keys()
Python 3 doesn't have dict.viewkeys method, renaming
to keys().

[YOCTO #9584]

(Bitbake rev: 8ae2fc26fc7c4f42817dece62b9a59fbda27c0b0)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-15 08:35:05 +01:00
Robert Yang 71765a90d1 bitbake: cooker.py: fix warn() -> warning()
Fixed:
DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead

(Bitbake rev: a3f464d202dafef4538e66c008cdecb7b8709ed1)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-13 12:16:10 +01:00
Richard Purdie f2e5d88461 bitbake: data_smart/utils: Add 'd' to the context used for better_eval in python expansion
If a line like:

foo=${@' '.join([d.getVar('D', True) + x for x in (' '.join([d.getVar('FILES_bash-' + p, True) or '' for p in ['lib', 'dev', 'staticdev', 'doc', 'locale', 'ptest']])).split()])}

is added to a function like do_install, it fails with Exception name 'd'
is not defined. This is due to a change of behaviour in python 3 compared
to python 2. Generator expressions, dict comprehensions  and set comprehensions
are executed in a new scope but list comprehensions in python 2.x are not. In
python 3 they all use a new scope.

To allow these kinds of expressions to work, the easiest approach is
to add 'd' to the global context. To do this, an extra optional parameter
is added to better_eval and we use that to add 'd'.

(Bitbake rev: 8f74881037bb01013d3d439dc0c269909a198c1c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-13 12:16:10 +01:00
Mariano Lopez 40e789d1fe bitbake: lib/bb/main.py: Fix use of BBPOSTCONF and BBPRECONF
The functionality of BBPOSTCONF and BBPRECONF was added in
commit 21b314d4d1 but there
was a typo in the variable name that raises an exception
in bitbake.

[YOCTO #9235]

(Bitbake rev: 6d1379c8818400e5cdc442e6142f08a110fd5b95)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-09 18:00:58 +01:00
Richard Purdie 438ac32628 bitbake: codeparser: Use hashlib for hashing, not hash()
"hash() is randomised by default each time you start a new instance of
recent
versions (Python3.3+) to prevent dictionary insertion DOS attacks"

which means we need to use hashlib.md5 to get consistent values for
the codeparser cache under python 3. Prior to this, the codeparser
cache was effectively useless under python3 as shown by performance
regressions.

(Bitbake rev: 12d43cf45ba48e3587392f15315d92a1a53482ef)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-03 13:48:33 +01:00
Richard Purdie 5f2facfc1d bitbake: codeparser: Small optimisation to stop repeated hash() calls
No functionality change, just avoids function call overhead in a
function which loops heavily.

(Bitbake rev: 633c0c19f87a92497a7e9771811cdc953e1b7047)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-03 13:48:33 +01:00
Paul Eggleton 02ac95ab5d bitbake: fetch2: fix unpacking of deb packages
deb packages in modern Debian versions have the data tarball compressed
with xz rather than gzip, and thus explicitly extracting data.tar.gz
fails. Unfortunately ar doesn't support wildcards matching items to
extract, so we have to find out what the name of the file is first and
then extract it, relying on tar to figure out how to unpack it based on
the filename rather than doing it with pipes and making that
determination ourselves.

(Bitbake rev: 17ff08d225a8fa7faffd683c028369574954fba9)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-03 13:13:31 +01:00
Peter Kjellerstedt 08c6808d01 bitbake: fetch2/git.py: References must match exactly
Previously the code used to match a reference to its SHA-1 in
_latest_revision() used the Python "in" operator, which made it match
if the reference matched the beginning of an existing tag or
branch. This test, however, must be exact. I.e., either the reference
matches a tag or branch exactly, or it does not match at all.

(Bitbake rev: e5986c78a6108fd7578989c20efcbf0b81c97e03)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-03 13:13:30 +01:00
Markus Lehtonen 2660ddeb2d bitbake: cookerdata: print an error if layer dir does not exist
Makes it easier for user to identify problems, e.g. typos, in BBLAYERS.

[YOCTO #9507]

(Bitbake rev: 32c9689e4b492dc5821749e284e397d717af2a6c)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-03 13:13:30 +01:00
Ed Bartosh 9434eee39a bitbake: Revert "buildinfohelper: work around unicode exceptions"
This commit causes buildinfohelper to crash when run on python 3
as python 3 doesn't have unicode builtin function.

This reverts commit 7a309d964a.

(Bitbake rev: 750ca5c8d5a25fc519b75c56352dec7823c7e240)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:04 +01:00
Ed Bartosh 2448df7d11 bitbake: xmlrpc: add parameter use_builtin_types
Added use_builtin_types parameter to XMLRPCProxyServer.__init__
to fix this error:
    ERROR: Could not connect to server 0.0.0.0:37132
    : __init__() got an unexpected keyword argument 'use_builtin_types'

(Bitbake rev: ceb6e5bd33a25c45c2afe1559b9394c466db8a92)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:04 +01:00
Ed Bartosh 0224f75999 bitbake: toaster: fix imports to work for python 3
Some APIs have been moved to other modules in python 3:
 getstatusoutput: moved from commands to subproces
 urlopen: moved from urllib2 to urllib.request
 urlparse: moved from urlparse to urllib.parse

Made the imports work for both python versions by
catching ImportError and importing APIs from different
modules.

[YOCTO #9584]

(Bitbake rev: 1abaa1c6a950b327e6468192dd910549643768bb)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:03 +01:00
Richard Purdie 8a93f5f32e bitbake: goggle/image-writer: Drop since bitrotting and no longer used
The upgrade to python3 is the final nail in the coffin for image-writer
and the goggle UI. Neither seem used or recieve patches and are based
on old versions of GTK+ so drop them, and the remaining crumbs support
pieces.

(Bitbake rev: ee7df1ca00c76f755057c157c093294efb9078d8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:03 +01:00
Richard Purdie 685628ebc4 bitbake: image-writer/goggle: Disable pygtkcompat problems
Disable the problematic gtk usage for use with pygtkcompat. The following
commit removes these tools/UIs entirely but we may as well leave this
piece in the history in case anyone does want a starting point for reusing
them.

(Bitbake rev: c53c7418d392452450352ca2175667dbdbd92401)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:03 +01:00
Jeremy Puhlman e80a8c25dc bitbake: lib/bb: Set required python 3 version to 3.4.0
get_context was added to mutliprocessing as part of 3.4.0

(Bitbake rev: 710351610e3ca4a1b61abc67564f84907e9b2f1c)

Signed-off-by: Jeremy Puhlman <jpuhlman@mvista.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:03 +01:00
Richard Purdie a15e538d50 bitbake: data_smart: Simplify ExpansionError exception
This seemingly convoluted syntax doesn't work in python3. Instead
use the chained exception handling syntax which appears to make more
sense here.

(Bitbake rev: b19a4c5166303b1fa680582adf63e6a5564bfb4c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:02 +01:00
Richard Purdie 0f2c59367a bitbake: bitbake: Convert to python 3
Various misc changes to convert bitbake to python3 which don't warrant
separation into separate commits.

(Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:02 +01:00
Richard Purdie ef1df51651 bitbake: bitbake: Drop futures usage since we're python 3
(Bitbake rev: bf25f05ce4db11466e62f134f9a6916f886a93d9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:02 +01:00
Richard Purdie 2192a48a5d bitbake: cooker: Fix parse progress for python3
Under python the type conversions can mean there are float values
used for triggering the parse progress events which then fails.
Add an explict int() conversion to ensure the parse events are
generated under python3.

(Bitbake rev: 138329c58e92744c56aae3ab70ceeef09613250c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-21 12:15:45 +01:00
Michael Wood db84307055 bitbake: toaster: Remove DATABASE_URL being passed around as an environment var
We don't need to pass the DATABASE_URL around and read it back if we
setup the django framework in the correct way.
We make the default sqlite database path a full path so that the
database isn't being assumed to be in CWD.

Also add some more useful comments on the database settings.

This is preparation work to migrate the build tests and be able to
trigger builds on differently configured databases.

(Bitbake rev: 973c740404ca6a09feea250d3433075995067fe0)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-20 10:09:02 +01:00
Richard Purdie c9867c4624 bitbake: cooker: Improve taskgraph file handling
Use open() instead of file() and close files when finished with them.

(Bitbake rev: 033c5a16ff19781ed793c2d97d285884017a2a4e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-19 15:19:11 +01:00
Richard Purdie 2970800a09 bitbake: codeparser: Increase cache version after code parser dependency changes
(Bitbake rev: 4d880f1d5794f569b6eb5f6e7e3001dd671633c0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-19 09:05:21 +01:00
Christopher Larson 1fb330bea6 bitbake: bb.codeparser: track variable flag references
Previously we only tracked the flags (minus excluded) of variables we depend
on, but not the flags we use explicitly.

(Bitbake rev: bdeb3dcd7c92e62a7c079e7b27048c4114f24a3a)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-19 09:05:21 +01:00
Christopher Larson 32ee30a46d bitbake: Provide LAYERDIR_RE for layer.conf
This variable is a regex-escaped version of LAYERDIR, for safer use in
BBFILE_PATTERN, so as to avoid issues with regex special characters in the
layer path.

[YOCTO #8402]

(Bitbake rev: 72900522778b6ff08b135bf8bb97dff3f1a20bd9)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-19 09:05:21 +01:00