We've been building to this for a while, default to return expanded
values for getVar/getVarFlags.
We can then go through and remove the "True" option to many of the
calls to this function, all function calls should have a default by now
though since the parameter has been required for a while.
(Bitbake rev: caf5bb9b7fe254bca9da077ebcb84a37d1f96dd4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a line like:
foo=${@' '.join([d.getVar('D', True) + x for x in (' '.join([d.getVar('FILES_bash-' + p, True) or '' for p in ['lib', 'dev', 'staticdev', 'doc', 'locale', 'ptest']])).split()])}
is added to a function like do_install, it fails with Exception name 'd'
is not defined. This is due to a change of behaviour in python 3 compared
to python 2. Generator expressions, dict comprehensions and set comprehensions
are executed in a new scope but list comprehensions in python 2.x are not. In
python 3 they all use a new scope.
To allow these kinds of expressions to work, the easiest approach is
to add 'd' to the global context. To do this, an extra optional parameter
is added to better_eval and we use that to add 'd'.
(Bitbake rev: 8f74881037bb01013d3d439dc0c269909a198c1c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This seemingly convoluted syntax doesn't work in python3. Instead
use the chained exception handling syntax which appears to make more
sense here.
(Bitbake rev: b19a4c5166303b1fa680582adf63e6a5564bfb4c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Various misc changes to convert bitbake to python3 which don't warrant
separation into separate commits.
(Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It seems the frozenset constructor in pypy runs len(), so we can't pass the
DataSmart instance directly to it, instead pass the iterator. Fixes pypy
support.
(Bitbake rev: b492836e08745e04bd9ba2fb0b56a680a5fdce79)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake variables don't include ":" characters so exclude these from the variable
expansion regexp.
This assists when parsing shell code which does A=${B:-C} as we don't want a
dependency on a variable called "B:-C".
(Bitbake rev: 14ed41a292123374d94f5c786a619881f2ddea42)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
At some point in the future, getVarFlag should expand by default. To
get there from the current position, we need a period of time where the
expand parameter is mandatory.
This patch starts that process. Clear errors will result from any code
which doesn't provide this. Layers can be fixed with an expression
like:
sed -e 's:\(\.getVarFlag([^,()]*, [^,()]*\)):\1, False):g' -i `grep -ril getVar *`
(Bitbake rev: aa3faebdf6af66ab34f74d328b2113de0b08c7ee)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
At some point in the future, getVar should expand by default. To get
there from the current position, we need a period of time where the
expand parameter is mandatory.
This patch starts that process. Clear errors will result from any code
which doesn't provide this. Layers can be fixed with an expression
like:
sed -e 's:\(\.getVar([^,()]*\)):\1, False):g' -i `grep -ril getVar *`
(Bitbake rev: fab717d303df0bcef737661f6917f275f35215a4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We don't expand anonymous python before execution, so nor should
we do this when calculating checksums for them.
(Bitbake rev: 5f10987edda35b08970a6dd6ccf9febad271ce3e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This sets the scene for removing the default False for expansion from
getVarFlag. This would later allow True to become the expand default.
On the most part this is an automatic translation with:
sed -e 's:\(\.getVarFlag([^,()]*, [^,()]*\)):\1, False):g' -i `grep -ril getVar *`
There should be no functional change from this patch.
(Bitbake rev: 7c3b99c6a716095af3ffce0b15110e91fb49c913)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If variables are unset, the code simply doesn't expand them, there
aren't errors. If the code is a python expression, this can get a bit
messy, see the attached test case. The python expansion code sees the }
of the unexpanded value rather than the close of the python expression
and then raises a SyntaxError exception.
Ideally, we'd update the code to match pairs of brackets. I don't know
how to do that with the current regex and this is unfortunately a
performance sensitive piece of code. We also run the risk of breaking
existing code in OE-Core where there are "{" characters but not "}"
to close them (PKGE and PE).
Rather than raising the exception, matching the existing "just return
the expression" behaviour seems more consistent with the standard
variable behaviour.
This addresses an issue found in the recent image.bbclass code where
there are some variables we choose not to expand (TMPDIR/DATETIME).
This patch also adds a test case for this behaviour. It wouldn't preclude
improved bracket matching code in the future either.
(Bitbake rev: d80d39e73223a50fda0090784303d2c57167bb4c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Our current OVERRIDES handling means we end up caching and checking for
a lot of possible override combinations which turn out to very unlikely.
A typical example is the SRC_URI variable where we have to check if
"URI" is an override. Having spent many hours working in this code, I've
realised all the actual overrides we use are lower case and our standard
variables are mostly uppercase.
This means we could gain quite some speed advantage if we write this
into the code, that overrides only consist of lowercase characters. This
patch shows how simple this is and the resulting speed gains are
significant. This is a significant change but tests show we don't appear
to have any users of capitals in overrides in any OE-Core metadata.
Before "time bitbake -p":
real 2m4.224s
user 7m32.312s
sys 0m7.116s
After "time bitbake -p":
real 1m26.009s
user 5m10.484s
sys 0m4.640s
This check could also be made conditional however I'm not seeing a need
to do that at present.
(Bitbake rev: c9b9443faa76ee7366b1400a56f826f3f9dec1be)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The dependencies of OVERRIDES were not including DEFAULTTUNE in OE-Core.
This is pulled in by a bb.utils.contains() reference which the override
dependency tracking code wasn't accounting for.
This patch ensures we do track contains references too.
(Bitbake rev: f3ee534cb0560dbb5f88a0ffe01e9305bae102e1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Right now, OVERRIDES dependency variables set using ??=, e.g. TARGET_ARCH
in OE-Core don't have their dependencies tracked. This is a bug, fix it.
(Bitbake rev: 944734503768f9e9223ef041f2d7873455418a54)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sadly its not enough to consider the dependencies of OVERRIDES, we
need to resolve their dependencies and so on recursively. If we don't
do this, some variable can be changed and the resulting data store is
incorrect.
(Bitbake rev: 82143ac064d391300e762ba7520ef1f8df18b574)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the values that make up OVERRIDES are themselves overridden,
we end up into some horrible circular logic. Unfortunately some
metadata does depend on this functionality.
e.g:
DEFAULTTUNE_virtclass-multilib-xxx = Y
which changes TUNE_ARCH
which changes TARGET_ARCH
which changes OVERRIDES
As a solution, we iterate override expansion until the values don't
change. If we iterate more than 5 times we abort and tell the user to
report the issue.
(Bitbake rev: 10279697c701e01bf6fdd5e9f92792ef5134807b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
No code changes, just refactoring to allow for functionality
changes by moving things to a separate function.
(Bitbake rev: 2eb934814179ccf42e3d424dabe26b17d013a7ed)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This gives us the needed context of the original ExpansionError, which is
invaluable when we have a chain of function calls in the expansion.
(Bitbake rev: c514b6fbea77ede1b7871b89592a33ed39b1d71c)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake-layers needs to map recipe and class files to the layer they
came from within the show-recipes and show-overlayed commands. However,
it turns out that mapping a file to the layer it came from is not as
trivial as it might seem. To do it properly we need to match the path to
an entry in BBFILES then map that to the collection name using
BBFILE_PATTERN, then map that to the actual layer using variable
history. If it doesn't match any entry in BBFILES, then we can fall back
to BBFILE_PATTERN (to handle classes and conf files).
This fixes the layer name not showing up properly in the output of the
show-recipes and show-overlayed commands for recipes in layers such as
meta-intel that have subdirectories in BBFILE_PATTERN. It also fixes the
priority not showing up in show-layers for such layers.
As part of this I've added a function to VariableHistory which for a
space-separated list variable gives you a dict mapping the items added
to the files in which they were added. I've also fixed
bb.utils.get_file_layer() and reduced some of the duplication by using
this function in bitbake-layers. Also fixes the priority not showing up
for layers such as meta-intel
Fixes [YOCTO #8160].
(Bitbake rev: e852f6cabd7489585477ab567a1afeb2252377ac)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Calling record() for each override alteration is slow. Since we now expand
overrides dynamically we don't have to record the log data at each alteration,
we can instead print it directly from the existing data stores at variable
history print time using the exact same data stores.
This massively improves performance of the data store when parsing
with bitbake -e for example, it will improve memory overhead as well.
The only downside is that VariableHistory has to poke into the datastore
for some of its data but that seems an acceptable tradeoff rather than
double caching.
(Bitbake rev: 100b447a161ef20fa559e39516cd32fa78e38262)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As things stand now, bitbake -e (which turns on all the caller tracking)
of OE-Core generates around 9.5 million stat calls which is slow and the
largest single thing on the profile data.
This is because infer_caller_details() calls traceback.extract_stack()
which adds line contents to the traceback. This in turn calls python's
internal linecache code which calls stat on every file for every callback.
We don't even use that info. We only even want a single frame of the stack.
Instead, open code for the pieces of information we need. Also, only
obtain the stack once for both halves of the infer_caller_details()
code.
This reduces the number of stat calls to around 0.5 million and significantly
improves parsing with bitbake -e.
(Bitbake rev: 7be76d8f79ea92fd4bd36146eb9a4b86551b526d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using deepcopy() caused a major performance regression. Turns
out we can work with a shallow copy as long as we force the
recreation of any list that we change, effectively a poor mans
CoW. This isn't too invasive and avoids the huge overhead of
deepcopy so this seems like the best way to have performance and
a working data store.
(Bitbake rev: 32dff4749c32f32e947c42c86f8357b8b607354b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
dbus sets RDEPENDS_dbus-ptest_class-target = "X". nativesdk as
BBCLASSEXTEND was expanding and setting various RDEPENDS variables
which were clearing overrides queued against RDEPENDS. The problem
was that this was leaking into other data contexts such as the
target case. This was because overridedata was a shallow copy.
Replacing the shallow copy with a deep copy avoids this problem of
leakage between the data stores at a small performance penalty.
(Bitbake rev: 83e2923cd3c8f2e25987132f85b06ce72bc941d7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that overrides get expanded 'on the fly', change appendVar
and prependVar to work using _append and _prepend, else we'd have
to re-implement pieces of getVar and the timing of expansions
becomes problematic.
Using _append/_prepend equivalence gives the behaviour users likley
expect from these functions.
(Bitbake rev: 40d661aaf7a563c6447b073310c5f2fdae6ca3d0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The expand cache can be cleared in the middle of getVar now
due to the use of operations like delVar. We therefore need
to check if variables are in the cache before accessing in
case it was cleared.
(Bitbake rev: a97ce216cfe44136f742383542954bfce027831e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
d.keys() was not returning items which had no base content but the
variable came into existance through overrides. We have to process
self.overridedata to find these other variables.
(Bitbake rev: 77fc57c610d7f65f7e328ad555f525b2f8bdf1d7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This performs better than continually regeneratiing the set().
Also only use set comparisions when its necessary to save some overhead
as issubset() is slower in the single item case.
(Bitbake rev: db9a4eedcf78969ec50bf0e4a8defba8ff0daa4a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When expanding OVERRIDES, its possible someone might try and override a variable
that is used in OVERRIDES. This could lead to infinite recursion. Add in
guards against this.
(Bitbake rev: 07d773369f571028c2cf82dd1f65d9731af6d00e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Updating the value of OVERRIDES whenever it changes turns out to be
extremely expensve/pointless. Instead, clear its value and re-establish
the value when we're going to use it.
This gives significant speed back.
(Bitbake rev: 41cf8d0c92d2d8a33fdad0921e424a0024914be1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using an internal flag for override mapping turns out to be slower
than is optimal, not least as we don't want the keys list to list
variables that have no value other than a potential override expansion.
Maintinaing a seperate cache is therefore more optimal from a speed
perspective.
(Bitbake rev: 1a39ccf66b30b54e1274669cb0e1dc8fc72dba49)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Trying to look up a variable called 'copy' in COW is problematic due
to internal implmentation details, at least avoid tracebacks from this.
Also don't duplicate override history (which is an atrefact of changed
override behaviour) as otherwise the bitbake -e output is convoluted.
(Bitbake rev: dddff5b7b8e6c18515b43389cef35503468b843d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When parsing we find problems if we clear prepends/appends when
setting variables during the initial parsing phases. Later, we actively
want to do this (in what would be post finalisation previously).
To handle this, pass a parsing flag to the operations to control
the correct behaviour for the context.
(Bitbake rev: ae87f5b8bf16191b3201cfb445062938eab992a0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Compeltely remove the replaces functionality and move all overrides
handling to getVar time. We can move the cookie cache into a hidden
flag within the variables themselves.
This removes the need for any of the internal_finalize steps.
This obsolete the need for the _seen_overrides COW cache.
(Bitbake rev: 2a0b73110bede91777ada54d1d89b45fb6034b6c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than repeatedly expanding the value of OVERRIDES, cache
the value and update it when things change.
There were also some bugs introduced in the replaces functionality
which this approach fixes by ensuring the replaces data is updated
at the correct points.
(Bitbake rev: f3b7c3e054ce230ea5c2db5813390383e8dfd6db)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These write operations should clear the expand cache since they can
influence returned variable values but currently do not. Fix this.
(Bitbake rev: a075332c9e13be35f1ae84adc6b29e9137a487ff)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than the heavy lifting in internal_finalize, move the bulk of
the functionality to getVar and rely on a new internal replaces variable
to keep track of any mappings that are needed. This removes the need
for the COW _special_values cache.
This change in functionality also implies we need to track any changes
not only to OVERRIDES but also any variable which OVERIDES depends upon.
Add code to handle this.
Explicitly list FILE as a value OVERRIDES depends upon since it doesn't
automatically get detected and is of key importance to OVERRIDES,
otherwise PN doesn't update when FILE changes.
(Bitbake rev: a6f1377ce3386d274882072d1ae6da3b1834149b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move the update_data functionality into internal data store operations
so the main finalize (update_data) call is a nop.
To make this work we need to call the internal finalization function
whenever OVERRIDES is changed to ensure values get updated correctly.
This has performance issues but the subsequant patches look into this.
(Bitbake rev: 546d9932d6c4413824319a9d780c0d77d2725f4a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Given these assignments:
TEST="a b c d"
TEST_remove = "b d"
TEST evaluates to "a c". However, if the _remove override is given as a
variable:
TEST="a b c d"
FOO = "b d"
TEST_remove = "${FOO}
TEST evaluates to "a b c d", because when FOO is expanded it isn't split into a
list.
Solve this by splitting all members of removeactive once they've been expanded.
[ YOCTO #7272 ]
(Bitbake rev: 207013b6dde82f9654f9be996695c8335b95a288)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These only have one instance created which means your subsequent datastores
can contain echos of previous ones. Obviously this is not the behaviour
we want/expect. It doesn't affect bitbake too badly as we only have one
datastore, it does massively potentially break our selftests though.
Thanks to Tim Amsell for pointing out the now obvious problem!
(Bitbake rev: 9facf3604759b00e8fe99f929353d46f8b8ba5cb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you copy the datastore, then delete a variable, it still shows up
in d.keys() when it should not. This patch addresses the issue.
(Bitbake rev: f28ee1bb03cb32d3757fbef67c9fbe143e3dadfa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The defaultval field is intended to be internal and the only use of that field
outside of data.py is to skip over it when iterating over a value's flags.
For clarity and convenience, rename the field to _defaultval so that it is
considered internal and not exposed through the data API.
(Bitbake rev: 2800958dadaa5c055ba21d52c98d842d360f0785)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we split variables only at whitespaces, a slipped in tab will render
a value unremovable.
(Bitbake rev: 9f171ea755644ecd9d2b3d7ed13bf8ec09ec917a)
Signed-off-by: Stefan Müller-Klieser <s.mueller-klieser@phytec.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you have:
FOO = "${bindir}/X Y"
FOO_remove = "${bindir}/X"
the expected result is "Y". Currently this doesn't work since the removed
expressions are not expanded first. This patch adjusts things so the
expressions are expanded before being processed for removal.
Also add a test to ensure this case continues to work.
[YOCTO #6624]
(Bitbake rev: 72a1ca4a104ccab73d6abcbd44db9c2636a58572)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the depths of time we were rather confused about naming. bb files
are recipes, the event to skip parsing them should be SkipRecipe,
not SkipPackage. This changes bitbake to use the better name but
leaves the other around for now. We can therefore start removing
references to it from the metadata.
(Bitbake rev: 98d9e6e0f514a7cb7da1d99bf4bd5602b89426d6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you try:
Y = ""
Y_remove = "X"
in OE-Core, bitbake will crash with a KeyError during expansion. The reason
is that no expansion of the empty value is attempted but removal from is it
and hence no varparse data is present for it in the expand_cache.
If the value is empty, there is nothing to remove so the best fix is simply
not to check for None but check it has any value.
Also add a test for this error so it doesn't get reintroduced.
(Bitbake rev: af3ce0fc0280e6642fa35de400f75fdbabf329b1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Unfortunately we've been neglecting to pay the correct tributes to the
cookie monster and hence the datastore is malfunctioning.
Currently tributes are only paid on the last part of a variable after
the last "_" character. We need to split by *all* "_" characters since
an override may contain the character.
This fixes the code so the correct number of tributes are made. Paradoxically
parsing appears to be faster after this change.
(Bitbake rev: d1c712fd3a59fa804e6fd451612c30487671f3a2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
FOO = "foo bar"
FOO_remove = "bar"
FOO_FOO = "${FOO} ${FOO}"
would show FOO_FOO = "foo foo bar" rather than the expected "foo foo".
This is actually a cache bug, this patch ensures the right value is
put into the cache. The preceeding patch adds a test case to ensure
we don't regress in future.
[YOCTO #6037]
(Bitbake rev: 2a80735183e8faa110b4c6d8d85c4707f28e03a1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie
Christopher Larson
Paul Eggleton
Ross Burton
Stefan Müller-Klieser