Commit Graph

204 Commits

Author SHA1 Message Date
Peter Kjellerstedt b1f09df0f6 bitbake: ConfHandler: Use the same regular expression for all variable names
When the regular expression for matching a variable name was amended
with allowing the ~ character as part of the variable name, this was
never done to the regular expression that matches export
lines. Similarly, the regular expression that was used for matching
unset variables also used the one without support for the ~ character.

This unifies the regular expressions. For good measures it also
corrects the regular expression used to match a variable flag name for
the unset command to match the one used when setting a variable flag.

(Bitbake rev: acd2fd74ed467dc85ec75d5d0815f43e493f29bf)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-13 09:43:40 +00:00
Richard Purdie f0d5eb39c3 bitbake: lib: Drop now unneeded update_data calls
Now that the datastore works dynamically we don't need the update_data calls
so we can just remove them. They're not actually done anything at all for
a while.

(Bitbake rev: 2300beb50333bb620013b058a7309e7f2042101d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-15 20:08:15 -08:00
Richard Purdie 631229df72 bitbake: event/ast: Add RecipeTaskPreProcess event before task finalisation
There are various pieces of code which need to run after the tasks are
finalised but before bitbake locks in on the task dependencies. This
adds such an event so dependency changes in anonymous python can
be accounted for and acted upon by these specific event handlers.

(Bitbake rev: 4dcd0e53f5ff4bf4f2d6cbdc51ff33a5f5f206af)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-20 11:52:52 +00:00
Ross Burton 573c646d4c bitbake: BBHandler: use with instead of open/close
This is more pythonic and can handle unclosed file warnings better
than the previous code structure.

(Bitbake rev: 50633012a64a3b5f0662145e29ff426374fb7683)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:24 +00:00
Ross Burton 9e63f81c78 bitbake: ast: remove BBVERSIONS support
BBVERSIONS is moderately horrible and it doesn't appear to be actually used by
anyone, so remove it to simplify the finalise codepaths.

(Bitbake rev: 0bb188f01e396052b127e170a25246d79a6d6741)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:10 +00:00
Joshua Lock ddaac5e4e3 bitbake: bitbake: remove True option to getVarFlag calls
getVarFlag() now defaults to expanding by default, thus remove the
True option from getVarFlag() calls with a regex search and
replace.

Search made with the following regex:
getVarFlag ?\(( ?[^,()]*, ?[^,()]*), True\)

(Bitbake rev: c19baa8c19ea8ab9b9b64fd30298d8764c6fd2cd)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:09 +00:00
Joshua Lock 1fce7ecbbb bitbake: bitbake: remove True option to getVar calls
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.

Search made with the following regex: getVar ?\(( ?[^,()]*), True\)

(Bitbake rev: 3b45c479de8640f92dd1d9f147b02e1eecfaadc8)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:09 +00:00
Markus Lehtonen 3658f6d477 bitbake: cookerdata/ast: Fail gracefully if event handler function is not found
[YOCTO #10186]

(Bitbake rev: 107c47c4e6de6a596cf1aeca5c18dbc1c5b44dc4)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-02 18:09:49 +01:00
Jérémy Rosen 0eb6d709b6 bitbake: ast/ConfHandler: Add a syntax to clear variable
unset VAR
will clear variable VAR
unset VAR[flag]
will clear flag "flag" from var VAR

(Bitbake rev: bedbd46ece8d1285b5cd2ea07dc64b4875b479aa)

Signed-off-by: Jérémy Rosen <jeremy.rosen@openwide.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18 10:06:26 +01:00
Richard Purdie 4cd5647f12 bitbake: cache/ast: Move __VARIANTS handling to parse cache function
Simple refactoring to allow for multiconfig support.

(Bitbake rev: 266b848da40904446eb1d084bbdc5307a9b45197)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18 10:06:26 +01:00
Richard Purdie ff6d458f9a bitbake: parse/ast, event: Ensure we reset registered handlers during parsing
When parsing, we should reset the event handlers we registered when
done. If we don't do this, parse order may change the build, depending
on what the parse handlers do to the metadata.

This issue showed up as a basehash change:

ERROR: Bitbake's cached basehash does not match the one we just generated (
/media/build1/poky/meta/recipes-core/meta/nativesdk-buildtools-perl-dummy.bb.do_unpack)!

This is due to the eventhandler in nativesdk.bbclass being run, despite
this .bb file not inheriting nativesdk.bbclass. The parse order was
different between the signature generation and the main multithreaded
parse.

Diffsigs showed:

bitbake-diffsigs 1.0-r2.do_unpack.sigbasedata.*
basehash changed from 887d1c25962156cae859c1542e69a8d7 to cb84fcfafe15fc92fb7ab8c6d97014ca
Variable PN value changed from 'nativesdk-buildtools-perl-dummy' to '${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}'

with PN being set by the event handler.

(Bitbake rev: 0219271d4130c1f4cf071c7577a4101c54c04921)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-15 08:35:07 +01:00
Richard Purdie 0f2c59367a bitbake: bitbake: Convert to python 3
Various misc changes to convert bitbake to python3 which don't warrant
separation into separate commits.

(Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:02 +01:00
Richard Purdie ef1df51651 bitbake: bitbake: Drop futures usage since we're python 3
(Bitbake rev: bf25f05ce4db11466e62f134f9a6916f886a93d9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:02 +01:00
Richard Purdie 654eadfa30 bitbake: bitbake: Update logger.warn() -> logger.warning()
python deprecated logger.warn() in favour of logger.warning(). This is only
used in bitbake code so we may as well just translate everything to avoid
warnings under python 3. Its safe for python 2.7.

(Bitbake rev: 676a5f592e8507e81b8f748d58acfea7572f8796)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-11 10:34:30 +01:00
Richard Purdie 99184d7879 bitbake: BBHandler/ast: Merge handMethod and handleMethodFlags
The functionality overlap between these two functions is significant and
its clearer to handle both things together since they are intimately
linked. There should be no behaviour change, just clearer code.

(Bitbake rev: 391aa4afc91be90d8d3ee47e1bf797d6ebe61a71)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:32:46 +00:00
Richard Purdie 6ae662acdc bitbake: parse/ast: Mark anonymous functions as python functions
Anonymous functions are python functions, set the variable
flags as such so we can detect them and avoid expansion where
needed.

(Bitbake rev: 1b303785c578bbae3a89be8d751d80fba860f62e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-04 23:47:49 +00:00
Richard Purdie 4628fe12e7 bitbake: lib/bb: Add expansion parameter to getVarFlag
This sets the scene for removing the default False for expansion from
getVarFlag. This would later allow True to become the expand default.

On the most part this is an automatic translation with:

sed -e 's:\(\.getVarFlag([^,()]*, [^,()]*\)):\1, False):g'  -i `grep -ril getVar *`

There should be no functional change from this patch.

(Bitbake rev: 7c3b99c6a716095af3ffce0b15110e91fb49c913)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-04 23:47:49 +00:00
Richard Purdie f8185ff9f8 bitbake: ast: Add filename/lineno to mapped functions
Where we add in mappings for EXPORT_FUNCTIONS, add dummy filename
and lineno data so ensure the assumption that all python functions
have this is correct.

(Bitbake rev: 547128731e62b36d2271c4390b3fee2b16c535dc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:35 +00:00
Richard Purdie c61c1eb26a bitbake: BBHandler: Improve IN_PYTHON_EOF handling
Now we're actively using the line numbers for other thins, having
magic values like IN_PYTHON_EOF causes problems, in particular, 32
bit overflow on 32 bit machines.

There is a neater way to signal eof to feeder(), just using an extra
parameter so use this instead and drop the IN_PYTHON_EOF magic values.

This has the added bonus that line numbers are then correct for
python functions at the end of files.

(Bitbake rev: e0f05871c2a6f1e86ae19ad343c7c6f822ddb67e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-22 00:02:05 +00:00
Richard Purdie 0019edc818 bitbake: ast/event/utils: Improve tracebacks to include file and line numbers more correctly
Currently bitbake tracebacks can have places where the line numbers are
inaccurate and filenames may be missing. These changes start to try and
correct this.

The only way I could find to correct line numbers was to compile as a
python ast, tweak the line numbers then compile to bytecode. I'm open
to better ways of doing this if anyone knows of any.

This does mean passing a few more parameters into functions, and putting
more data into the data store about functions (i.e. their filenames
and line numbers) but the improvement in debugging is more than worthwhile).

Before:
----------------
ERROR: Execution of event handler 'run_buildstats' failed
Traceback (most recent call last):
  File "run_buildstats(e)", line 43, in run_buildstats(e=<bb.build.TaskStarted object at 0x7f7b7c57a590>)
NameError: global name 'notexist' is not defined

ERROR: Build of do_patch failed
ERROR: Traceback (most recent call last):
  File "/media/build1/poky/bitbake/lib/bb/build.py", line 560, in exec_task
    return _exec_task(fn, task, d, quieterr)
  File "/media/build1/poky/bitbake/lib/bb/build.py", line 497, in _exec_task
    event.fire(TaskStarted(task, logfn, flags, localdata), localdata)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 170, in fire
    fire_class_handlers(event, d)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 109, in fire_class_handlers
    execute_handler(name, handler, event, d)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 81, in execute_handler
    ret = handler(event)
  File "run_buildstats(e)", line 43, in run_buildstats
NameError: global name 'notexist' is not defined

----------------

After:
----------------
ERROR: Execution of event handler 'run_buildstats' failed
Traceback (most recent call last):
  File "/media/build1/poky/meta/classes/buildstats.bbclass", line 143, in run_buildstats(e=<bb.build.TaskStarted object at 0x7efe89284e10>):
         if isinstance(e, bb.build.TaskStarted):
    >        trigger = notexist
             pn = d.getVar("PN", True)
NameError: global name 'notexist' is not defined

ERROR: Build of do_package failed
ERROR: Traceback (most recent call last):
  File "/media/build1/poky/bitbake/lib/bb/build.py", line 560, in exec_task
    return _exec_task(fn, task, d, quieterr)
  File "/media/build1/poky/bitbake/lib/bb/build.py", line 497, in _exec_task
    event.fire(TaskStarted(task, logfn, flags, localdata), localdata)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 170, in fire
    fire_class_handlers(event, d)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 109, in fire_class_handlers
    execute_handler(name, handler, event, d)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 81, in execute_handler
    ret = handler(event)
  File "/media/build1/poky/meta/classes/buildstats.bbclass", line 143, in run_buildstats
    trigger = notexist
NameError: global name 'notexist' is not defined
----------------

(Bitbake rev: 1ff860960919ff6f8097138bc68de85bcb5f88b0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 12:18:19 +00:00
Richard Purdie 7db88aa67d bitbake: parse: Don't try to expand __base_depends/__depends
Trying to expand a variable which isn't a string doesn't make sense.

(Bitbake rev: 62367cca1f1793eb9827406bcdd5980fdeb80a60)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-11-16 11:55:07 +00:00
Christopher Larson 383450c78a bitbake: build: delete tasks thoroughly
We want addtask to be able to bring back a deleted task, but we don't want its
previous dependencies to come back with it, so rather than marking a task as
deleted and then skipping tasks marked as such, actually delete the task and
its dependency information in deltask.

While we're in that part of the code, also fix a couple 'not foo in bar'
instances to 'foo not in bar', which is preferred in python.

(Bitbake rev: 94b3f3d6bdfbfa47f7eb3c3de64940a145b2ddd1)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-03 12:33:08 +01:00
Paul Eggleton 9b05ef581c bitbake: lib/bb/parse: properly handle OSError when updating mtime cache
If a file no longer exists, drop it from the cache silently instead of
generating a traceback. This was visible in some cases when a recipe was
deleted when bitbake was resident in memory.

(Bitbake rev: fe105b9042bdac4afd9f38fcf92bfdc2c04ec23f)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-19 18:05:48 +01:00
Christopher Larson 22078d5e53 bitbake: bb.parse: properly error out on filesystem errors
We've had a long-standing bug where a legitimate error reading a file (IOError
or OSError) is always suppressed as though it was a 'file not found' case. As
a concrete example, if you do a `chmod 000 conf/local.conf`, it'll silently
not parse local.conf, rather than erroring to let the user know about the
problem.

Fix this by handling the ENOENT case specifically.

(Bitbake rev: e691312a3add222b04e7b2f52f8df6abcb9068bf)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-01 22:24:20 +01:00
Richard Purdie 933e19697b bitbake: parse/ast/data_smart: Add parsing flag to getVar/setVar
When parsing we find problems if we clear prepends/appends when
setting variables during the initial parsing phases. Later, we actively
want to do this (in what would be post finalisation previously).

To handle this, pass a parsing flag to the operations to control
the correct behaviour for the context.

(Bitbake rev: ae87f5b8bf16191b3201cfb445062938eab992a0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-07-12 22:50:45 +01:00
Richard Purdie 69b6919341 bitbake: bitbake: Add explict getVar param for (non) expansion
Rather than just use d.getVar(X), use the more explict d.getVar(X, False)
since at some point in the future, having the default of expansion would
be nice. This is the first step towards that.

This patch was mostly made using the command:

sed -e 's:\(getVar([^,()]*\)\s*):\1, False):g' -i `grep -ril getVar *`

(Bitbake rev: 659ef95c9b8aced3c4ded81c48bcc0fbde4d429f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-06-23 11:57:53 +01:00
Richard Purdie ffaa0f1b8a bitbake: parse/BBHandler: Avoid repeatedly resetting FILE
If we're not going to change the value of FILE, or we know it isn't
going to have changed (ext == bbclass), don't set FILE.

This avoids messy looking history of the variable as well as optimises
parsing speed slightly.

(Bitbake rev: 88e4600aa66dda2e6c807f9d97af8982bcd8817b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-06-05 16:42:01 +01:00
Richard Purdie 3db9d6e02d bitbake: ConfHandler: Fix bogus dependency problems
Adding a dependency when errors occur accessing a file when calling
handle() is not the correct thing to do. THe handle() code calls
resolve_file() which can raise an exception without ever touching
"fn" itself, it has also already marked all the dependencies correctly.

This leads to bogus files being resolved to the local cwd and
hence triggers reparses for no good reason.

The solution is to simply remove the bogus dependency.

(Bitbake rev: 366af3be1cffd64e4a79c15990c1e05869022c14)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-05-29 10:17:16 +01:00
Richard Purdie 91c45c0163 bitbake: BBHandler: Drop cwd from search path
Whilst bitbake has done this for a long time, the behaviour of resolving
class files against cwd is not desirable. This can be seen during
base configuration parsing when looking for base.bbclass where a dependency
on cwd is added. If cwd then changes, the cache is invalid and triggers a
re-parse.

The only real option is to drop this entry and if files can't be found, we
fix BBPATH in the cases where it needs fixing. I didn't find any in the
random selection of layers I tested parsing locally.

(Bitbake rev: 508aad9d5db7e51328b1fd6ee53b4bc3720a30b7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-05-29 10:17:16 +01:00
Richard Purdie fd731142e9 bitbake: BBHandler: Error for incomplete function definitions
Add some sanity checks on the parsing state engine when returning data
so that incomplete functions raise parse errors.

This means a recipe doing:

do_somefunction {
     echo 1

VAR = "1"

will now raise a ParseError. To get the right file/line information,
__infunc__ was changed to a list.

[YOCTO #7633]

(Bitbake rev: 6b54a72638f57882d4fd5aab96b2752a09e065af)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-05-15 17:45:00 +01:00
Pascal Bach 087424d925 bitbake: parse/ast: Fix issue if path contains '&'
(Bitbake rev: 4fea138f7cef53626a40decb96207dbaf9284020)

Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-01-29 22:11:35 +00:00
Richard Purdie 2a6f739f1d bitbake: ConfHandler: Clean up bogus imports
The import statements here are plain bizarre. Remove them, tweaking
some of the function calls to match current practices. I can't find any
reason these old imports are as they are.

(Bitbake rev: 4c2f1fe51a13ddc97e518327714292af46b9e1ab)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-01-16 08:25:30 +00:00
Richard Purdie 5c9e2b2f43 bitbake: ConfHandler: Rename oldfn to parentfn to be clearer
Looking at this function I had no idea what oldfn was, I doubt anyone
else would either without looking up what the caller does. "parentfn"
would seem a more appropriate name so rename it.

(Bitbake rev: fc70ed596703a1aa954223b169d4ad51193a6ec1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-01-16 08:25:30 +00:00
Richard Purdie 6c3c3e11f6 bitbake: cooker/cache/parse: Implement pyinofity based reconfigure
Memory resident bitbake has one current flaw, changes in the base configuration
are not noticed by bitbake. The parsing cache is also refreshed on each invocation
of bitbake (although the mtime cache is not cleared so its pointless).

This change adds in pyinotify support and adds two different watchers, one
for the base configuration and one for the parsed recipes.

Changes in the latter will trigger a reparse (and an update of the mtime cache).
The former will trigger a complete reload of the configuration.

Note that this code will also correctly handle creation of new configuration files
since the __depends and __base_depends variables already track these for cache
correctness purposes.

We could be a little more clever about parsing cache invalidation, right now we just
invalidate the whole thing and recheck. For now, its better than what we have and doesn't
seem to perform that badly though.

For education and QA purposes I can document a workflow that illustrates this:

$ source oe-init-build-env-memres
$ time bitbake bash
[base configuration is loaded, recipes are parsed, bash builds]
$ time bitbake bash
[command returns quickly since all caches are valid]
$ touch ../meta/classes/gettext.bbclass
$ time bitbake bash
[reparse is triggered, time is longer than above]
$ echo 'FOO = "1"' >> conf/local.conf
$ time bitbake bash
[reparse is triggered, but with a base configuration reload too]

As far as changes go, I like this one a lot, it makes memory resident bitbake
truly usable and may be the tweak we need to make it the default.

The new pyinotify dependency is covered in the previous commit.

(Bitbake rev: 0557d03c170fba8d7efe82be1b9641d0eb229213)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-01-14 11:32:02 +00:00
Richard Purdie 2e26745a13 bitbake: ast: Add error when trying to use dash in sh function names
A dash character is illegal in function names in sh (but not bash). Since
our shell tasks run under sh and the shell parser is sh based, EXPORT_FUNCTIONS
won't work with class names containing a dash.

We can't change sh, we can ensure the user is warned about the problem
straight away though.

[YOCTO #7006]

(Bitbake rev: 86704281b79e524dccccc88cbf996b299b33bae2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-01-08 10:38:36 +00:00
Ross Burton b92ec90ea4 bitbake: data: rename defaultval to _defaultval
The defaultval field is intended to be internal and the only use of that field
outside of data.py is to skip over it when iterating over a value's flags.

For clarity and convenience, rename the field to _defaultval so that it is
considered internal and not exposed through the data API.

(Bitbake rev: 2800958dadaa5c055ba21d52c98d842d360f0785)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-12-03 12:24:00 +00:00
Richard Purdie 91a164cb87 bitbake: event: Add SkipRecipe event to replace SkipPackage
In the depths of time we were rather confused about naming. bb files
are recipes, the event to skip parsing them should be SkipRecipe,
not SkipPackage. This changes bitbake to use the better name but
leaves the other around for now. We can therefore start removing
references to it from the metadata.

(Bitbake rev: 98d9e6e0f514a7cb7da1d99bf4bd5602b89426d6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-06-01 14:29:31 +01:00
Richard Purdie 56cd778eb0 bitbake: parse: Improve file inheritance logging
The file inheritance logging has been here since the dawn of time. It
duplicates output many times over and logs to debug level 2. When running
with the debug option, its understandable the user may want to see the
paths of files included in the build.

These changes remove pointless/duplicate output and print the
include paths clearly at debug level one in a form which
users should be able to more easily parse.

(Bitbake rev: 633a56987df639295fe9128418da5634050063e5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-05-11 13:15:12 +01:00
Richard Purdie bc391938c4 bitbake: parse/ast: Show append logging at lower log level
It was reported that bitbake -D made no mention of which append files it
was using. bitbake -DD does but it makes sense to increase the log level
of this piece of debug information.

[YOCTO #6262]

(Bitbake rev: 5824bf9c6feea05567d155911f4ab2e371911d34)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-05-11 12:27:56 +01:00
Richard Purdie 249a24eee0 bitbake: parse/ast: Optimise data finalisation
The optimisation where only the data we're interested in was finalised
was good but it turns out we can do better. In the case where a
class-extension is to be targeted, we can skip the other targets.

This change does that and speeds up parsing at the bitbake-worker
execution time. Specifically, you can see an improvement in the speed
of bitbake X -n.

(Bitbake rev: b56918c7ef7913e84356c69ee9b269844a446728)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-04-23 11:43:29 +01:00
Marius Avram 888683de7a bitbake: ConfHandlerr: Use full path in variable history
When an evaluation was made for a configuration file the path to the
file was saved as a relative one. The change in this commit will save the
location as an absolute path. This way the user will have full information
regarding the location of the file where a variable was changed and the
line withing the file.

[YOCTO #5562]

(Bitbake rev: df9e22901555b06fef308f7136547f2c47ccec35)

Signed-off-by: Marius Avram <marius.avram@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-03-21 13:51:51 +00:00
Jacob Kroon f5a344441a bitbake: ast: Fix support for anonymous methods in wildcard .bbappend files
When using wildcard .bbappend files with anonymous methods in them,
bitbake/python fails to parse the generated code since the '%' is encoded
in the generated method name.

Fix this by including '%' in the convert-to-underscore list during
method name mangling.

While we're at it, move the method name mangling translation table
to a class variable, as suggested by Chris Larson.

[YOCTO #5864]

(Bitbake rev: 537f1f9bbe110acc9848ef95f43468c07d87af79)

Signed-off-by: Jacob Kroon <jacob.kroon@mikrodidakt.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-02-24 12:42:40 +00:00
Paul Eggleton a56bd812a7 bitbake: parse: make vars_from_file return None for non-recipes
It doesn't really make sense to set PN from .conf files, for example.
More concretely, this avoids the config hash changing unnecessarily
within Hob due to PN effectively changing (since bblayers.conf is
parsed first and then .hob.conf).

(Bitbake rev: 22e03ef7ac9bb6b7245250347ae9c10c19f1d74e)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-02-17 15:34:45 +00:00
Richard Purdie ba82035412 bitbake: build/ast: Create strong task add/del API in bb.build
Currently its near impossible to control task addition/deletion from
metadata context. This adds stong add/deltask API to bb.build
which is traditionally where it resided. The rather broken
remove_tasks function was removed, it didn't appear to do anything
useful or have any users.

This allows us to clean up hacks currently in use in metadata and use
standard API for it instead.

(Bitbake rev: bf7138dd38fc1f8efca80891198e3422fef64093)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-12-18 17:08:07 +00:00
Richard Purdie 8a82a3835c bitbake: ast/BBHandler/build: Add support for removing tasks (deltask)
Back in the depths of time we did support task removal. In the pre
AST days it was nearly impossible to continue supporting it, it wasn't
used so it was dropped. With the modern codebase we can easily now support
deltask and it would be very useful within the metadata since it can
massively simplify dependency trees.

As an example, a core-image-sato had 47703 inter task dependencies before
this patch and a patch to native.bbclass, afterwards with the noexec tasks
deleted, we had 29883. Such a significant simplification is worthwhile
and justifies adding a deltask operation to the system.

(Bitbake rev: acecbde6fb70ff3c96deab3cdf819d8442e87ed4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-12-18 17:08:07 +00:00
Richard Purdie f5f3ef7b09 bitbake: parse/ConfHander/BBHandler/utils: Fix cache dependency bugs
Currently bitbake only adds files to its dependency list if they exist.
If you add 'include foo.inc' to your recipe and the file doesn't exist,
then later you add the file, the cache will not be invalidated.

This leads to another bug which is that if files don't exist and then
you add them and they should be found first due to BBPATH, again the
cache won't invalidate.

This patch adds in tracking of files we check for the existence of so
that if they are added later, the cache correctly invalidates. This
necessitated a new version of bb.utils.which which returns a list of
files tested for.

The patch also adds in checks for duplicate file includes and for now
prints a warning about this. That will likely become a fatal error at
some point since its never usually desired to include a file twice.

The same issue is also fixed for class inheritance. Now when a class
is added which would be found in the usual search path, it will cause
the cache to be invalidated.

Unfortunately this is old code in bitbake and the patch isn't the
neatest since we have to work within that framework.

[YOCTO #5611]
[YOCTO #4425]

(Bitbake rev: 78d285871e4b8c54ccc4602d571e85f922e37ccd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-11-30 22:26:30 +00:00
Martin Jansa aa0b237144 bitbake: parse: Return IOError when including file with absolute path
* resolve_file was behaving different when relative and absolute
  paths were passed to it

* include relative-path/non-existent-file.inc
  works correctly resolve_file throws IOError, BBHandler.py:handle()
  doesn't catch it, ConfHandler.py:include() catches IOError and shows:
  DEBUG: CONF file 'relative-path/non-existent-file.inc' not found
* include /absolute-path/non-existent-file.inc
  was failing, because resolve_file just returns fn,
  BBHandler.py:handle() calls bb.parse.mark_dependency(d, abs_fn)
  which throws:
  OSError: [Errno 2] No such file or directory: '/absolute-path/non-existent-file.inc'
  and parsing fails.
  Ad isfile() test for absolute fn and throw IOError to make
  resolve_file behavior consistent for both paths.

* I know we had some issues with -b relative-path-to-recipe.bb and
  absolute path, so consider this patch only as RFC and documentation of
  this problem

* Catch OSError too in ConfHandler.py:include() e.g. in case the file exists, but user
  cannot read it or something like that.

(Bitbake rev: b0bbd89a4f0b98fa1ab28b8e0526cd9ddb76fa57)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-08-26 11:29:46 +01:00
Li Wang 10e44f162c bitbake: bitbake: python funcname can not include special character @
[YOCTO #4772]

When path:file change to python function, it maybe include '@' character.
So, add the special character to change to '_' for avoid error.

(Bitbake rev: 684bc6dcb11ecb1fd7a4d25c08909ad9879e8342)

Signed-off-by: Li Wang <li.wang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-06-27 10:06:09 +01:00
Bogdan Marinescu 5aa5fc470e bitbake: bitbake: Add event mask flag support
Add a flag to event handlers which lists the events a given handler wishes to
process. By default event handlers recieve all events but this means
we can stop running code in many cases if we know it doesn't want the event.

This is part of the fix for YOCTO #3812, but implements filtering only
for class event handlers; the other part (events filter for UIs) will be
the subject of a different patch.

(Bitbake rev: 074003a4e7530a72863b9c685fc5c31b0f08c039)

Signed-off-by: Bogdan Marinescu <bogdan.a.marinescu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-06-14 12:52:58 +01:00
Richard Purdie 8bef99d3c8 bitbake: methodpool: Retire it, remove global method scope
Having a global method scope confuses users and with the introduction
of parallel parsing, its not even possible to correctly detect conflicting
functions. Rather than try and fix that, its simpler to retire the global
method scope and restrict functions to those locations they're defined
within. This is more what users actually expect too.

If we remove the global function scope, the need for methodpool is reduced
to the point we may as well retire it. There is some small loss of caching
of parsed functions but timing measurements so the impact to be neglibile
in the overall parsing time.

(Bitbake rev: 4d50690489ee8dc329a9b0c7bc4ceb29b71e95e9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-05-24 10:34:52 +01:00