Commit Graph

374 Commits

Author SHA1 Message Date
Richard Purdie 0f2c59367a bitbake: bitbake: Convert to python 3
Various misc changes to convert bitbake to python3 which don't warrant
separation into separate commits.

(Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:02 +01:00
Richard Purdie ef1df51651 bitbake: bitbake: Drop futures usage since we're python 3
(Bitbake rev: bf25f05ce4db11466e62f134f9a6916f886a93d9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:02 +01:00
Richard Purdie 2192a48a5d bitbake: cooker: Fix parse progress for python3
Under python the type conversions can mean there are float values
used for triggering the parse progress events which then fails.
Add an explict int() conversion to ensure the parse events are
generated under python3.

(Bitbake rev: 138329c58e92744c56aae3ab70ceeef09613250c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-21 12:15:45 +01:00
Richard Purdie c9867c4624 bitbake: cooker: Improve taskgraph file handling
Use open() instead of file() and close files when finished with them.

(Bitbake rev: 033c5a16ff19781ed793c2d97d285884017a2a4e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-19 15:19:11 +01:00
Christopher Larson a809f1c6af bitbake: bb.cooker: show limited traceback for parsing ExpansionError
It's useful to see tracebacks for ExpansionErrors, but only if we skip the
leading bitbake-internal elements, otherwise we see elements of the expansion
process.

As one example:

Before:

    ERROR: ExpansionError during parsing /scratch/yocto-new/external-as-needed/poky/meta/recipes-core/glibc/glibc-locale_2.23.bb: Failure expanding variable PV[:=], expression was ${@get_external_libc_version(d)} which triggered exception AttributeError: 'module' object has no attribute 'external'

After:

    ERROR: ExpansionError during parsing /scratch/yocto-new/external-as-needed/poky/meta/recipes-core/glibc/glibc-locale_2.23.bb
    Traceback (most recent call last):
      File "PV[:=]", line 1, in <module>
      File "/scratch/yocto-new/external-as-needed/meta-sourcery/recipes-external/glibc/glibc-external-version.inc", line 3, in get_external_libc_version(d=<bb.data_smart.DataSmart
    object at 0x7f05d2566950>):
             sopattern = os.path.join(d.getVar('base_libdir', True), 'libc-*.so')
        >    found_paths = oe.external.find_sysroot_files([sopattern], d)
             if found_paths:
    ExpansionError: Failure expanding variable PV[:=], expression was ${@get_external_libc_version(d)} which triggered exception AttributeError: 'module' object has no attribute 'external'

(Bitbake rev: 7ff5b9eed82b7f4fd138fc6d746a0b79efbea98a)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-19 09:05:20 +01:00
Richard Purdie 471310e5e7 bitbake: cooker/toasterui: Drop SEND_DEPENDS_TREE UI feature
Now the event is sent unconditionally we can drop this feature
as its no longer needed.

(Bitbake rev: 473deeb0fc6065693e1fcfcbb8b79753103db537)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-14 23:05:14 +01:00
Richard Purdie 7c0a219d29 bitbake: cooker: Fix log message syntax
Ensure we pass the string parameter correctly.

(Bitbake rev: 7ed82bd1fe7bdd93b0614119c42eb218dc5d83e6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-13 13:41:32 +01:00
Christopher Larson c6394a56f7 bitbake: bb.{cooker, data}: only emit a var as python if 'func' is set
This avoids a common issue where PACKAGECONFIG is emitted as a function in
bitbake -e when the 'python' flag exists. It isn't a python function unless
both 'func' and 'python' are set. This aligns with the behavior of
emit_func_python.

(Bitbake rev: c5e45063cb3ae17bbe3304ea5e712bd76e686c4a)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-06 10:22:18 +01:00
Richard Purdie 8e7282c032 bitbake: cooker/knotty: Prefix parse logs with filename being parsed
We now prefix log messages coming from worker task context with the
PF and task info, however parsing messages all have to be manually
prefixed which is ugly and error prone. This change modifies the log
handler filter so this happens automatically, meaning we don't have
to change every message to include that information. This makes error
messages longer but more usable.

(Bitbake rev: 1af0ccaac81e182c4ca520037dda362d180e5605)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-15 06:59:44 +01:00
Richard Purdie 1131303ec4 bitbake: cooker: pass exception to finishAsyncCommand
An invalid task causes bitbake to exit incorrectly, firing a
CommandCompleted event rather than a CommandFailed one. This
means that clients listening for CommandFailed events are
unable to detect the build failure even though one occurred.

Passing an exception string to finishAsyncCommand when a task
fails causes the CommandFailed event to be fired correctly.

[YOCTO #9087]

(Bitbake rev: 98a2c37e077b16e3bc8bb102bd18b293130d15a4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-15 06:59:44 +01:00
Markus Lehtonen 2554be49f2 bitbake: cooker: fix CookerParser.shutdown()
Prevent a hang when shutdown() is called during parsing (e.g. after
SIGINT). We must not append 'None' to the jobs queue. Otherwise the
worker loop inside Parser.realrun() may break out at the wrong point,
causing the results queue thread blocking bitbake indefinitely.

[YOCTO #9319]

(Bitbake rev: 7ebea3e9a60232222efa8a546a0ff28a53029949)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-31 09:13:04 +01:00
Richard Purdie 1ca71e5178 bitbake: cooker: Ensure bbappend order is deterministic
Currently bbappend files in a layer are applied in the order they're
found on disk (as reported by glob) which means things are not
deterministic.

By sorting the glob results, the order becomes deterministic, the parsing
order for .bb files also should be deterministic as a result of this change.

[YOCTO #9138]

(Bitbake rev: 3f8febc4212fbd3485ac9bdd4ac71b8fb0a05693)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-30 12:01:29 +01:00
Markus Lehtonen 97617fd675 bitbake: bb/cache: drop some unused arguments
Drop unused 'd' argument from the cache save methods, simplifying the
API.

(Bitbake rev: 81bc1f20662c39ee8db1da45b1e8c7eb64abacf3)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:41:15 +00:00
Richard Purdie 7593756cbb bitbake: cooker: Improve cache handling
We're seeing issues where the self test, which uses tinfoil doesn't
notice the changed contents of include files. The issue is
cached_statements in the parser being reused when the files have changed.

Whilst looking at this, I realised there were some other issues:

* We need to also invalidate the mtime cache when cooker restarts
* We should pass full filenames to the file invalidation code
* We should process cached_statements as part of inotify invalidation

With these fixes, the caching is more reliable for memory resident
bitbake too. It does raise some questions about cache validation and
lifecycles and indicates bitbake does need more work in the area,
preferably with the removal of the globals. This at least highlights
and works around some of the current issues.

(Bitbake rev: 3f507ff8bc467fba936cf3f31bb8ea8e02f168e8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-15 16:48:12 +00:00
Ed Bartosh f9f93ae856 bitbake: cooker: gracefully shutdown parsers
CookerParser.shutdown code doesn't do all required work to shutdown
parser processes. As a result bitbake hangs if interrupted during
parsing. Putting None into the parser_quit queue should fix this issue
as it makes parsers to quit main loop.

(Bitbake rev: f67307977e8f089ce6d208d3e9de2a6a1768757e)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11 22:34:19 +00:00
Richard Purdie aece748762 bitbake: build/data: Don't expand python functions before execution [API change]
Right now, if you have some python code like:

X = "a"

def somefunction(d):
    d.setVar("X", "b")
    d.setVar("Y", "${X}")

then any sane person would expect that Y = "b" at the end of the
function. This is not the case, Y = "a".

This is due to the python function being expanded before execution, the
executed code would read d.setVar("Y", "a"). This understandably
confuses people, it also makes it near impossible to write ${} in a
python function without unintended things happening.

I think there is general agreement we should fix this and standardise
on non-expansion of python functions. We already don't expand anonymous
python (mostly).

I've checked OE-Core with buildhistory before and after this change and
there were a small number of issues this exposed which I've sent
patches for.

I propose we default to not expanding python code and then deal with
any consequences from that if/as/where identified. This will improve
new user understanding and usability of the system, it also allows
several long standing weird expansion issues to be fixed.

(Bitbake rev: 8bf33a8e92c0e188fa392030025756196c96fcbb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 15:47:11 +00:00
Richard Purdie e39cfb1f9c bitbake: cooker: Don't expand python functions in variable dumps
We don't want to expand python functions since they aren't expanded
at execution time (e.g. anonymous python). They can also have side
effects.

This function is primarily used by toaster for variable dumps for later
display. The lack of expansion of python functions won't matter in this case
and actively helps some variable handling (e.g. SRCPV).

(Bitbake rev: 3f5520b4844a4bdd615046479ba08ed192bdc8cd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 15:47:11 +00:00
Schroeder, Henning f8a44b160d bitbake: cooker: extended dot styling
Extended the dot styling of dependencies created by bitbake -g in dot syntax to differentiate between the various kinds.
  depends: solid
  rdepends: dashed
  rrecommends: dotted

The change observed is that depends get an explicit style which is the same as dot default behavior and the runtime recommends get
dotted while before they were dashed. This helps to distinguish them graphically as well as eases post processing by script.

(Bitbake rev: 86e78e0ca7aa5452411f35239942ecee3d8824ec)

Signed-off-by: Henning Schroeder <henning.schroeder@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:32:46 +00:00
Ed Bartosh 916c73dfef bitbake: cooker: shutdown cooker parser on shutdown
Currently any not catched exception in cooker causes bitbake
to hang because of not terminated children of CookerParser.

Long term solution would be to reimplement Cooker as a context
manager and terminate parser children in its __exit__ method.

Partial fix is to call CookerParser.shutdown in Cooker.shutdown in
hope that all Cooker exceptions are caught and shutdown method is
called.

[YOCTO #8900]

(Bitbake rev: 3f67600dc3292bc8208644ce89e8bf7ab95cf2e7)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-06 23:13:47 +00:00
Phil Blundell a4a5d1f323 bitbake: cooker, bitbake-worker: Fix spelling of "received"
I before E, except after C...

(Bitbake rev: 14c9593265f7469cb8a205a46f845ac7491246df)

Signed-off-by: Phil Blundell <pb@pbcl.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-01 11:20:42 +00:00
Richard Purdie 8f6b9c75a7 bitbake: cooker: Only start as many parse threads as we need
If we're only going to parse one recipe, no point in starting
a large number of threads.

(Bitbake rev: b977faf59dc08050a44a16032fe52d1bbb80f2a1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-01 11:20:42 +00:00
Richard Purdie 5375e6431c bitbake: bitbake: Set process names to be meaninful
This means that when you view the process tree, the processes
have meaningful names, aiding debugging:

$ pstree -p 30021
bash(30021)───KnottyUI(115579)───Cooker(115590)─┬─PRServ(115592)───{PRServ Handler}(115593)
                                                ├─Worker(115630)───bash:sleep(115631)───run.do_sleep.11(115633)───sleep(115634)
                                                └─{ProcessEQueue}(115591)

$ pstree -p 30021
bash(30021)───KnottyUI(117319)───Cooker(117330)─┬─Cooker(117335)
                                                ├─PRServ(117332)───{PRServ Handler}(117333)
                                                ├─Parser-1:2(117336)
                                                └─{ProcessEQueue}(117331)

Applies to parse threads, PR Server, cooker, the workers and execution
threads, working within the 16 character limit as best we can.

Needed to tweak the bitbake-worker magic values to tell the
workers apart.

(Bitbake rev: 539726a3b2202249a3f148d99e08909cb61902a5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-30 11:43:56 +00:00
Peter Kjellerstedt a948f5252a bitbake: cooker: Allow BBMASK to contain multiple regular expressions
Before, BBMASK was only permitted to contain one regular expression.
This made it hard to add to the BBMASK in multiple places as one was
supposed to separate the different regular expressions with a "|"
rather than with whitespace as is customary in BitBake variables.

Now one can specify any number of regular expressions in BBMASK. This
makes it possible to, e.g., mask out recipes in another layer from the
layer.conf file.

This also properly ignores any regular expressions that do not compile
(before an invalid regular expression would cause a ParseError in the
first bbappend file found stating that it was not a BitBake file...)

(Bitbake rev: 2c778ad50aceaffb855baf5f4aa0fed98c880870)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-29 17:03:53 +00:00
Paul Eggleton 76a281c870 bitbake: taskdata: add the ability to access world targets list
In certain circumstances it can be useful to get access to the world
targets list from a recipe in order to add dependencies on some or all
of the items in it. If a special function, 'calculate_extra_depends' is
defined in the recipe, and the recipe is to be built, then call it at
the right point before we calculate which tasks should be run. The
function can append items to the "deps" list in order to add
dependencies. This is not as tidy a solution as I would have liked, but
it does at least do the job.

As part of this change, the buildWorldTargets function was moved to
bb.providers to make it possible to call from taskdata.

Part of the implementation of [YOCTO #8600].

(Bitbake rev: aba0dce57c889495ec5c13919991a060aeff65d2)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-22 12:45:44 +00:00
Ed Bartosh f2b7252190 bitbake: cooker: add providermap to dep_tree
Added providermap information to the result of buildDependTree API.

This will be used by Toaster to map virtual dependencies to recipes.

(Bitbake rev: d3e07368549f30265f59846a260efa8230a225ca)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:26:32 +00:00
Ross Burton bfa7859ffa bitbake: cooker: fix findFilesMatchingInDir documentation
The documentation for findFilesMatchingInDir() was inconsistant with the
implementation: the regex was escaped before searching so effectively it's a
pure textual substring, and the machine example was broken.

(Bitbake rev: 6bef981488ec94b46dbe3797acfecf9c4b6ecbbc)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-08 13:58:08 +00:00
Ross Burton 3d4273716d bitbake: cooker: use in instead of count
No point counting all instances when we just want to know if there's any or not.

(Bitbake rev: 3369072efb653339da8dbd1ca864ff8e1ff899ca)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-08 13:58:08 +00:00
Ed Bartosh 01419d5565 bitbake: cooker: add state.get_name method
Added method to convert state code into the human readable name.
It will be used in logging and error reporting.

(Bitbake rev: 9ec6379b27d210214d0b3f2e55962f721b7f5f51)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:34 +00:00
Richard Purdie 2c9431135d bitbake: cooker: Drop useless parsing exception
The SyntaxError exception simply shows the recipe that failed to parse
which is pretty useless without the actual exception. We could make it
print more info, however we can just use one of the more generic handlers
instead and remove this one.

For a python indentation error, this leads to a much more readable error
message.

(Bitbake rev: 9241eb10847634e34c5ff8767ed8c114f66ff6cf)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:34 +00:00
Richard Purdie 3ebf7617d6 bitbake: cooker: Ensure BB_CONSOLE remains correct over server resets
The console log data is written to is created at console initialisation
time and does not change over reset events. This ensures the
BB_CONSOLELOG value is correct over such resets by preserving it.

(Bitbake rev: 335eb2db228f7543a49de71f063ac72b865c947a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-29 07:35:52 +00:00
Ed Bartosh 4ab7202c31 bitbake: cooker: preserve pre and post configs
Additional config files passed to bitbake server with --read and
--postread options are rewritten by client bitbake even if
it doesn't use those options.

This is a show stopper for toaster as toaster command line
builds are based on the assumption that server is aware of
toster configs, provided by --postread option.

This behaviour is fixed by preserving values of --read and
--postread options when bitbake server starts and restoring
them if client bitbake doesn't explicitly specify them.

(Bitbake rev: 02c64f7487ca8ec5d32c440f5002c4b8f64b76a6)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-27 07:24:30 +00:00
Ed Bartosh 6fa3fecb12 bitbake: cooker: normalize build targets
BuildStarted event not fully represents build tasks for
the targets. If -c option is used to specify default task
it's not included into the event.

Made build targets to always look as <target>:do_<task>.
Consider default task (do_build or specified by -c command
line option) when normalizing.

(Bitbake rev: 0b0e214e6f53c97ad3d48f622c7fc0ca149956f6)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-16 14:13:23 +01:00
Alex Franco 7ef2f951d6 bitbake: cooker/runqueue: Allow bitbake commands starting with do_
The output of "bitbake, -c listtasks pkg" lists tasks with their real names
(starting with "do_"), but then "bitbake -c do_task" fails, as "do_" always
gets unconditionally prepended to task names. This patch handles this error
by checking whether a task starts with "do_" prior to prepending it with it
when the task runlist is being constructed (and a few other corner cases).

[YOCTO #7818]

(Bitbake rev: dd3050ceef37ac556546e940aa596ce96ef6c8df)

Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-04 16:23:58 +01:00
Paul Eggleton 294948c732 bitbake: lib/bb/cooker: add interrupted flag to BuildCompleted event
Allow any listeners for this event (such as buildhistory.bbclass in
OpenEmbedded) to find out if the build was interrupted rather than
completing normally. The value will be 0 if not interrupted, 1 if
interrupted waiting for remaining tasks to complete, or 2 if force
interrupted (stopping any running tasks immediately).

(Bitbake rev: df2b778efd2ecc48f6c5a3ed446f6459f2250035)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-29 14:03:49 +01:00
Paul Eggleton 3c3ff92882 bitbake: cooker: drop appendlist
Now we have the bbappends list and all users have been converted over to
use it, we don't need this anymore.

(Bitbake rev: 279770c42d4c63aa2cebce331b55a92a564b50ac)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-19 18:05:46 +01:00
Paul Eggleton b84bf58c19 bitbake: cooker: drop appliedappendlist
Whilst collecting this list on the fly may be quicker, doing so within a
function that's meant to *query* the list of bbappends is poor practice.

(Bitbake rev: 5c12aa3b0010d7d1733e54a0ca7d0af465454210)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-19 18:05:46 +01:00
Paul Eggleton 7ce27b9fce bitbake: cooker: ensure prefile/postfile can work in memory resident mode
The prefile/postfile options weren't working in memory resident mode
because they weren't being passed through to the server, so ensure that
they do get passed through and that the server is reset when the values
come through.

(Bitbake rev: a3f7dc042fc7b1c308bfd248431930eb8ba50326)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-19 18:05:45 +01:00
Richard Purdie 37a2c062a5 bitbake: command: ensure sync commands that read configuration see updates
Add a means of ensuring that synchronous commands that read the results
of the configuration trigger a reparse of the configuration if any
underlying files have changed.

(Bitbake rev: aaf3cc024315450c1674819edf2a4e5cdf293f35)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-19 18:05:45 +01:00
Paul Eggleton 6e5ac6ba8e bitbake: cooker: further limit inotify watches
Unfortunately we were acting on inotify notifications about any files
changing within the watched directories, not just the ones we actually
care about. In OE the build directory is in BBPATH and hence it gets
watched, and we write things like bitbake.lock and
bitbake-cookerdaemon.log to that directory, hence effectively
notifications were being tripped on every bitbake invocation. To avoid
this, record which file/subdirectory we're interested in against each
watched directory so we can ignore any events for files/subdirectories
we don't care about.

Additionally, if we move up to the parent dir, ensure we haven't already
seen it before adding a watch on it (we were previously calling
watcher.add_watch() on the same directory multiple times before in a
typical OE configuration).

(Bitbake rev: bc39b8da34c046b629c43fd0a8cac2efbf1c060f)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-19 18:05:44 +01:00
Paul Eggleton 715d857174 bitbake: Fix default function parameter assignment to a list
With python you should not assign a list as the default value of a
function parameter - because a list is mutable, the result will be that
the first time a value is passed it will actually modify the default.
Reference:

http://docs.python-guide.org/en/latest/writing/gotchas/#mutable-default-arguments

(Bitbake rev: 7859f7388f2e3f675d0e1527cfde18625f36f637)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-19 18:05:44 +01:00
Richard Purdie a8b723498c bitbake: cooker: Resolve file monitoring race issues when using memres bitbake
If you load memory resident bitbake, then change bblayers.conf or other
configuration files, then execute your first command, bitbake will not
notice the changes. This is because it adds the file watches during
inital parsing, which happens at the time the first command is run.

To fix this, we move the addition of the file watches to earlier in
the process, so bitbake then does track them correctly. This also
avoids some issues Paul was seeing with tinfoil2.

(Bitbake rev: b9375c73e736003e66575969c9ea244403e47aeb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-03 07:36:25 +01:00
Richard Purdie 4c2f28bf48 bitbake: cooker: properly fix bitbake.lock handling
If the PR server or indeed any other child process takes some time to
exit (which it sometimes does when saving its database), it can end up
holding bitbake.lock after the UI exits, which led to errors if you ran
bitbake commands successively - we saw this when running the PR server
oe-selftest tests in OE-Core. The recent attempt to fix this wasn't
quite right and ended up breaking memory resident bitbake. This time we
close the lock file when cooker shuts down (inside the UI process)
instead of unlocking it, and this is done in the cooker code rather than
the actual UI code so it doesn't matter which UI is in use. Additionally
we report that we're waiting for the lock to be released, using lsof or
fuser if available to list the processes with the lock open.

The 'magic' in the locking is due to all spawned subprocesses of bitbake
holding an open file descriptor to the bitbake.lock. It is automatically
unlocked when all those fds close the file (as all the processes terminate).
We close the UI copy of the lock explicitly, then close the server process
copy, any remaining open copy is therefore some proess exiting.

(The reproducer for the problem is to set PRSERV_HOST = "localhost:0"
and add a call to time.sleep(20) after self.server_close() in
lib/prserv/serv.py, then run "bitbake -p; bitbake -p" ).

Cleanup work done by Paul Eggleton <paul.eggleton@linux.intel.com>.

This reverts bitbake commit 69ecd15aece54753154950c55d7af42f85ad8606 and
e97a9f1528d77503b5c93e48e3de9933fbb9f3cd.

(Bitbake rev: a29780bd43f74b7326fe788dbd65177b86806fcf)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-01 07:34:09 +01:00
Richard Purdie 61fbff2cd4 bitbake: cooker/taskdata: Make NoProvider errors non-fatal to -e/-g options
If you have a situation where you are getting a "Nothing PROVIDES" error
(for example when something you request to build DEPENDS on something
that has been skipped or doesn't exist) it would be useful to be able to
use bitbake -g or bitbake -e to debug it, but currently both of those
are blocked by the error.

This patch adds an "allowincomplete" option to taskdata and uses this
for the -e/-g bitbake options. The NoProvider errors are still
printed and bitbake does return an error exist code but the environment
and task graph files are generated.

[YOCTO #7623]

(Bitbake rev: 2ca36a9f088438a8d1db44119c704f9480b04298)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-07-12 22:50:41 +01:00
Leonardo Sandoval 7bccf59a24 bitbake: cooker.py: Lock/Unlock members function into BBCooker
(Bitbake rev: d66dccf9f9a33bfef5c28cc1c767bfc89faee532)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-07-09 18:00:21 +01:00
Richard Purdie 25702f3a2f bitbake: cooker: Improve DATE/TIME handling
Currently, once set, DATE and TIME in the metadata remain unchanged.
This is suboptimal for cases where cooker is reused. This ties setting
the variables into the BuildStarted event which seems like a more
appropriate time to do so.

It also changes BUILDNAME to be based off DATE/TIME by default if not
already set so that the data is more consistent. We therefore need to
expand the value rather than the previous default of not doing so.

This change does mean the date/time values are in sync across all
variables too.

It does mean bitbake now has special knowledge of DATE/TIME but that
would seen unavoidable (other than doing this in event handlers which
has its own set of downsides).

[YOCTO #5187]

(Bitbake rev: f883cf240266ee7be2cbd8971a8164cf4df9e372)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-07-08 00:06:08 +01:00
Richard Purdie 00ca43be05 bitbake: cooker: Fire BuildCompleted before finishing the command
Some handlers hook on BuildComplete so it avoids certain event races
to finish the command after the BuildComplete event is sent out.

This means the UI is available to handle events until the command
completes.

What appears to be a race on one of the sanity tests for event handlers
triggered this change although the failure is hard to reproduce.

[YOCTO #7921]

(Bitbake rev: e42d7c47a06fbb5981e0313478c8e3656b99f4e7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-06-26 09:27:33 +01:00
Richard Purdie 69b6919341 bitbake: bitbake: Add explict getVar param for (non) expansion
Rather than just use d.getVar(X), use the more explict d.getVar(X, False)
since at some point in the future, having the default of expansion would
be nice. This is the first step towards that.

This patch was mostly made using the command:

sed -e 's:\(getVar([^,()]*\)\s*):\1, False):g' -i `grep -ril getVar *`

(Bitbake rev: 659ef95c9b8aced3c4ded81c48bcc0fbde4d429f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-06-23 11:57:53 +01:00
Richard Purdie 4bedfeffa2 bitbake: cooker/event: Add an event which lists all stamps reachable after parsing
The metadata can potentially use such an event to clean up any
"unreachable" data, solving several problems we currently have
where obsolete data may continue to exist in the shared areas.

(Bitbake rev: c5e6f929f3d5eeb7954660dea62611c58b795ff8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-06-09 13:30:09 +01:00
Richard Purdie 08b77c8cef bitbake: cooker/utils: Improve parsing profiling
Currently the cooker parsing processes each dump an individual profile
which is ok, but means absolute numbers of function calls for a given load
can be tricky to determine as parsing of recipes may go to different pool
threads on different runs.

This change collects up the individual thread parsing results and processes
them into one profile output. The profile processing function in utils
needed tweaks to allow this to work.

(Bitbake rev: d3d2541aacd1ea560da0d8b25a3ea3f0563dee70)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-05-29 10:17:16 +01:00
Ed Bartosh db409697db bitbake: bitbake: reset build mtime cache before the build
Introduced build mtime cache structure. Reset it before the build
to prevent bitbake from crashing when build/tmp/stamps hierarchy
is removed.

[YOCTO: #7562]

(Bitbake rev: f8590547a198a78334debdf14bf40acb50c22ecc)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-04-29 10:59:54 +01:00