In Hob settings, there is a tab to add/remove extra settings. This
patch implements a way to "remove" variables from conf files, through
bitbake. But, to keep the history assigment of the variables synchronized,
instead of removing, it replaces the lines with blank lines.
[YOCTO #5284]
(Bitbake rev: bd720fb63cef6b399619b8fbcaeb8d7710f2d6df)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code is happily trying to expand variable names containing newlines,
spaces and tabs which are illegal characters in variable names. This
patch stops it doing this. This will change dependency checksums
since some rather weird dependencies were being attempted to be expanded.
(Bitbake rev: 37e13b852b33d98fa40f49dc1e815b3bbe912ff0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The change to use the expansion cache in VariableParse was incorrect as
it was adding in references it shouldn't have been. This patch corrects
the codepaths and ensures the references are correct.
The cache version is bumped since the previous bug could have leave
to invalid checksum calculations and a clean cache is therefore desireable.
The impact of the bug was that sstate was not getting reused when it should
and some tasks were also being rerun when they should not have been.
(Bitbake rev: 8a42d082315bd6ce091d006bf83476db257fa48b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a variable references another but it isn't set at present, the
reference wasn't stored. It really should be marked as a reference
and the higher level dependency code can handle as appropriate.
(Bitbake rev: b05b748b2153c941b95cd36fb22aaafc4dbf3791)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow a list of flags to expand to be passed into getVarFlags. This
is useful within bitbake itself to optimise performance of the
dependency generation code.
(Bitbake rev: a3ae7efdf750fc5bb9ff5a75defbcfdab1912dbe)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Simple if xxx checks end up calling len(xxx). We're interested in the specific case
of None which means we can break out the iterator much earlier after the first
item. This adds in the specific tests for None in what is a hot path in the
data store code which gives small performance gains.
(Bitbake rev: a4d81e44a7cd3dafb0bf12f7cac5ff511db18e60)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Compute a cache of the list of potential export variables so
that we don't have to compute the list from scratch.
(Bitbake rev: f41f46f7eaa6889edeb3a4e4ddedc07084686c60)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When in VariableParse, use the expand_cache if possible rather than looking
up data. Ultimately it would come from the same place but this short cuts
a heavily used code block for speed improvements.
(Bitbake rev: f682b8b83d21d576160bac8dc57c4c989b4dc555)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Debugging showed the variable expansion regexp was catching python
expressions (starting with @). Since these are caught by their own
dedicated regexp, stop matching these for the plain variable expansion
for small performance improvements.
(Bitbake rev: c630d564285f55f9db10c18269bd310df797430e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake wasn't reparsing when _remove items were added to its configuration
and equally, appends/prepends were also being badly tracked. This
change enrures these variables are accounted for in the configuration
hash.
[YOCTO #5172]
(Bitbake rev: 62914f9208ef2427a34daa523af857f4027900eb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
DISTRO_FEATURES_remove = "opengl" wasn't working as expected. The reason
turned out the be the indirect reference to opengl and the fact _remove was
operating on unexpanded data.
This patch rearranges some code to ensure we operate on expanded data
by moving the expand cache handing into getVarFlags instead of getVar.
(Bitbake rev: 181899bd9665f74f8d1b22d2453616ad30d26d9e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is more idiomatic, and from the limited performance testing I did, is
faster as well. See https://gist.github.com/kergoth/6360248 for the naive
benchmark.
(Bitbake rev: 1aa49226d5a2bac911feeb90e3d9f19529bc1a3e)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are long standing complaints about the fact its very difficult
to remove a portion of a variable. The immediate request is for a -=
and =- operator. The trouble is that += and =+ are "immediate"
operators and are applied straight away. Most people would expect
-= and =- to be deferred to have the effect most people desire and
therefore implementing -= and =- would just make the situation more
confusing.
This deferred operation is much more similar to the override syntax
which happens at data store finalisation. The _remove operator is
therefore in keeping with the _append and _prepend operations.
This code is loosely based on a patch from Peter Seebach although it
has been rewritten to be simpler, more efficient and avoid some
potential bugs.
The code currently only works on space delimited variables, which
are by far the most commom type. If bitbake is ehanced to support
types natively in future, we can adjust this code to adapt to that.
(Bitbake rev: 9c91948e10df278dad4832487fa56888cd58d187)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(aka pay the cookie monster for weak defaults)
If you have code like:
MYVAR = "a"
MYVAR_override ??= "b"
then MYVAR will get the value "a" even when override is in OVERRIDES. The
reason is that the value of ??= is set as a flag not a value and the cookie
monster isn't paid.
The fix is to ensure appropriate payment is made for a defaultval varflag
matching the usual setVar case.
(Bitbake rev: 3d8044bc79c482c5ea008ddf12a8128dcd1527ee)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if the flags set against a variable in the base data store
change, it doesn't automatically trigger a reparse when it really
should. For example with the blacklist class setting:
PNBLACKLIST[qemu] = "bar"
PNBLACKLIST[bash] = "foo"
will not trigger a reparse if only one entry is changed and a
blacklisted recipe can still be built.
I did consider using BB_SIGNATURE_EXCLUDE_FLAGS in here however it
doesn't make sense, we want to trigger a reparse when any of the
flags change too (which is different to the sstate signatures which
we wouldn't want to change in those cases).
[YOCTO #4627]
(Bitbake rev: ed74ea50043f6feb698c891e571feda2b9f8513d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Take __BBTASKS, __BBHANDLERS and __BBANONFUNCS into account when
computing the configuration hash.
[YOCTO #4447]
(Bitbake rev: 260ced7452405fc43ce3d9dd6798236aa07cc716)
Signed-off-by: Laurentiu Palcu <laurentiu.palcu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Added a new command in bitbake to save a variable in a file; added a function
in cooker which is called by this command.
Added new command in bitbake to enable/disable data tracking.
The function saveConfigurationVar from cooker.py saves a variable in the file that
is received by argument. It checks all the operations made on that variable, using the history.
If it's the first time when it does some changes on a variable,it comments the lines where
an operation is made on it, and it sets it in a line to the end of file. If it's not
the first time(it has a comment before), it replaces the line.
Made some changes in hob to save the variables from bblayers.conf and local.conf
using the bitbake command.
[YOCTO #2934]
(Bitbake rev: 55b814ccfa413d461d12956896364ab63eed70a8)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds tracking of the history of variable assignments.
The changes are predominantly localized to data_smart.py and
parse/ast.py. cooker.py and data.py are altered to display the
recorded data, and turn tracking on for the bitbake -e case.
The data.py update_data() function warns DataSmart.finalize()
to report the caller one further back up the tree.
In general, d.setVar() does what it used to do. Optionally,
arguments describing an operation may be appended; if none
are present, the operation is implicitly ignored. If it's
not ignored, it will attempt to infer missing information
(name of variable, value assigned, file and line) by examining
the traceback. This slightly elaborate process eliminates a
category of problems in which the 'var' member of the keyword
arguments dict is set, and a positional argument corresponding
to 'var' is also set. It also makes calling much simpler for
the common cases.
The resulting output gives you a pretty good picture of what
values got set, and how they got set.
RP Modifications:
a) Split from IncludeHistory to separate VariableHistory
b) Add dedicated copy function instead of deepcopy
c) Use COW for variables dict
d) Remove 'value' loginfo value and just use 'details'
e) Desensitise code for calling order (set 'op' before/after
infer_caller_details was error prone)
f) Fix bug where ?= "" wasn't shown correctly
g) Log more set operations as some variables mysteriously acquired
values previously
h) Standardise infer_caller_details to be triggered from .record()
where at all possible to reduce overhead in non-enabled cases
i) Rename variable parameter names to match inference code
j) Add VariableHistory emit() function to match IncludeHistory
k) Fix handling of appendVar, prependVar and matching flag ops
l) Use ignored=True to stop logging further events where appropriate
(Bitbake rev: f00524a3729000cbcb3317fee933ac448fae5e2d)
Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This code adds inclusion history to bitbake -e output, showing
which files were included, in what order. This doesn't completely
resolve timing questions, because it doesn't show you which lines
of a file were processed before or after a given include, but it
does let you figure out what the path was by which a particular
file ended up in your build at all.
How it works: data_smart acquires a .history member, which is an
IncludeHistory; this represents the inclusion of a file and all its
inclusions, recursively. It provides methods for including files,
for finishing inclusion (done as an __exit__), and for
dumping the whole tree.
The parser is modified to run includes inside a with() to push
and pop the include filename.
RP Modifications:
a) Split Include and Variable tracking
b) Replace deepcopy usage with dedicated copy function
c) Simplify some variable and usage
(Bitbake rev: b2dda721262da8abb7dc32d019e18fbc32ed8860)
Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If SkipParse is raised from something which isn't anonymous python, it wasn't
being handled correctly. This improves the handling for example from within inline
python.
(Bitbake rev: 7467d7d66b24cc8f43ab168e65895e7c4aee6092)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
An issue was uncovered where changing:
IMAGE_INSTALL_append = "X"
to
IMAGE_INSTALL_append = "X Y"
in local.conf would not get noticed by bitbake. The issue is that
the configuration hash doesn't account for overrides or key expansion.
This patch improves get_hash to account for these. This means the hash
does account for changes like the above.
[YOCTO #3503]
(Bitbake rev: 86bf1f5603e8f98019544e45f51bd0db9a48112a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of logging the function/variable separately as a NOTE when
failing to expand, re-raise ExpansionError with more contextual
information. This means that the full details are reported in Hob as
well as actually reporting the original error message in any UI where
we previously did not. For example, we used to get this with tab/space
indentation issues in a python function:
NOTE: Error expanding variable populate_packages
ERROR: Unable to parse /path/to/recipename.bb
Now, we will get this:
ERROR: ExpansionError during parsing /path/to/recipename.bb: Failure
expanding variable populate_packages: IndentationError: unindent does
not match any outer indentation level (<string>, line 4)
Fixes [YOCTO #3162].
(Bitbake rev: ce5c7a95a359cdaecab7c4a519ad4f9df029da82)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If this regular expression is unanchored, it would accept strings like:
do_install_append1
do_install_appendsomelongstring
and treat them like they were do_install_append. Clearly this isn't desirable.
Only one instance of this type of issue was found in OE-Core and has been fixed
so correcting the regexp should be safe to do.
(Bitbake rev: 23bd5300b4a99218a15f4f6b0ab4091d63a602a5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Variables which used multiple overrides and the append/prepend operators were
not functioning correctly. This change fixes that.
This fixes the testcase:
OVERRIDES = "linux:x86"
TESTVAR = "original"
TESTVAR_append_x86 = " x86"
TESTVAR_append_x86_linux = " x86+linux"
TESTVAR_append_linux_x86 = " linux+x86"
[YOCTO #2672]
(Bitbake rev: dc35a2e506e15fb7ddbf74c3b3280e9e83ab33bb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the variable name is not specified then don't confuse the error message
by starting off with "Failure expanding variable None...".
(Bitbake rev: 9cb16f3c73751e7cf6d495586a6193f06eb97b1f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For config hash, we put the keys in structure of "set()", which is not
order sensitive. Therefore when calculating the md5 value for config
hash, we need to identify the order of the keys.
(Bitbake rev: 0f1b142a3f6b8125bf023c2e5ec269618869abf7)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adopt the BB_HASHCONFIG_WHITELIST as a exclusion list for variables that are
not needed in cache hash calculation.
(Bitbake rev: ae8cf138b5eb8f1f28a7143b8d67ad06cbe43061)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
CC: Christopher Larson <kergoth@gmail.com>
CC: Martin Jansa <martin.jansa@gmail.com>
CC: Andreas Oberritter <obi@opendreambox.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously we use the file time stamp to judge if a cache is valid.
Here this commit introduce a new method, which calculates the total
hash value for a certain configuration's key/value paris, and tag
it into cache filename, for example, bb_cache.dat.xxxyyyzzz.
This mechanism also ensures the cache's correctness if user
dynamically setting variables from some frontend GUI, like HOB.
(Bitbake rev: 1c1df03a6c4717bfd5faab144c4f8bbfcbae0b57)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds appendVar and prependVar functions to the data store
meaning python code would no longer have to do the getVar, append and
the setVar dance that much of the current python code does.
It also adds corresponding variants for flags.
Currently there is no spacing added by these functions. That could be
added as a parameter if desired.
If these functions turn out to be hotspots in the code, there are tricks
that could potentially be used to increase the speed of these specific
operations within the datastore.
(Bitbake rev: 4a4046268f84b85559eea2c4b6a6004ad8cccb77)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the vardeps flag is not None, we now silence the warnings about
non-literal usage for that variable.
(Bitbake rev: e724b9f417d1baf898f5afc6376c73c1a2ad8db9)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- If a name is passed to the parser, prepend the messages with "while
parsing <name>:". This gives a bit more context.
- Tweak the warning messages slightly (they had to be altered anyway to
inject the variable being parsed).
Before:
DEBUG: Warning: in call to 'bb.data.getVar': argument ''%s' % var' is \
not a literal
After:
DEBUG: while parsing emit_pkgdata, in call of bb.data.getVar, argument \
''%s' % var' is not a string literal
(Bitbake rev: 1060193ae4d54e667735dbff5d1d2be49a3f95c9)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if passing expand=True to getVar() function, it will pass the
handling to getVarFlag(), which doesn't get any benefit from the expand
cache.
Call the expand() function separately in getVar() to make use of the
expand cache, which can decrease the parsing time by 40%.
(from current 49s to 27s)
(Bitbake rev: 6555a77c199f41bf35460138764e03e30c56d29f)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As the code stands, setting a variable with ??= could result in a ?=
variable not overriding it. This patch fixes the issue by allowing
the ast to make lookups that ignore any ??= set variables.
(Bitbake rev: 32fee2e650dfdd3aa9a7572dad1251e0c24ca34b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we delete a variable we no longer expect it to override other
variables.
To do this we remove it from the list of active overrides at deletion
time. It turns out we already had to do this at override expansion time
so this cleans up the code to be more consistent as an added bonus.
(Bitbake rev: d924ff9ede57c3dea6e1c738ba3633f18d460b14)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you d.delVar(), you expect the variable to be gone. Even empty
variables continue to exist in the datastore and are still user visible
unfortunately. The COW siutation means you can't just remove it
since it might unmask a variable from an inner copy.
This patch therefore stops empty variables from appearing in key lists
exposed to the external world making empty variables an internal
implementation detail only.
(Bitbake rev: 2b5548c591d4cfde9238d2cc0959c42cfc08f09c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since we're going to creat the seen set() anyway, we might as well use
it directly. If we don't do this, we see thousands of function calls
with associated overhead on profiles.
(Bitbake rev: 9d43e3279895639ee4899df635f2546c7ee13737)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if you do:
OVERRIDES = "z"
DEPENDS_prepend = "a "
DEPENDS = "b"
DEPENDS_z = "c"
d.update_data()
d.getVar("DEPENDS")
gives "a c"
d.update_data()
d.getVar("DEPENDS")
then gives "c"
This patch changes the behaviour such that at the time bitbake expands the DEPENDS_z
override, it removes "DEPENDS_z" from the data store. In the above example this would
mean that it wouldn't matter how often you call d.update_data(), you'd always get
"a c" back.
See the bitbake-devel mailing list for further discussion and analysis of the
potential impact of this change.
(Bitbake rev: 899d45b90061eb3cf3e71029072eee42cd80930c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If expanding a variable triggers an exception the caller currently has no
way to supress the error message or otherwise handle the siutation. An
example of where this is a problem is "bitbake -e" showing tracebacks and
errors for variables like SRCPV in OE/Poky.
Secondly in a chained expansion fails, log mesages are recorded for
every step of the expansion, not just the innermost error which is
where the real failure occured.
To fix this we introduce a new exception ExpansionError which callers
can handle as appropriate.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if a variable has been set with ??= and the code looks it up
before the data finalisation phase, no value is found. This is causes
serious problems for anonymous python functions which manipulate data, or
for the fetcher revision handling code where revisions can be set with
??=.
There is also a significant performance implication for processing lazy
assignment in finalise.
Moving the check for a default value into getVarFlag addresses both
the timing issue and the performace. This change gives a 7% real time
performance improvement to parsing the Poky metadata. The cost of the
check at this point is minimal since we have all the data flags available.
This should also fix Yocto bug 752.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 53740977521bc81ffa37adfa7bbeb8f2a80ea165)
build: write logfiles per task, not per function
Based on d14f9bf6 from poky, reworked for master and other cleanup.
(Bitbake rev: beadff2eca1eb95f0411115dd72ddb4c3c44c604)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Cristiana Voicu
Richard Purdie
Alexandru DAMIAN
Christopher Larson
Laurentiu Palcu
Peter Seebach
Paul Eggleton
Ross Burton
Dongxiao Xu
Richard Purdie