SIGINT should be from the user, not a script. It also doesn't work as
reliably to shut down processes, as it's not always interpreted as a
termination request. In addition, it causes KeyboardInterrupt exceptions in
the worker processes, which can interfere with our exception handling.
(Bitbake rev: e5f6e0e9de4c6d1dfdd269d2bf7f83c00c415a27)
Signed-off-by: Chris Larson <clarson@kergoth.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Bug 606 report that if $DL_DIR is read-only, do_fetch will
simply hang without any error message.
The root cause is that: bb.fetch.go()->bb.utils.lockfile()
will try to lock file ${DL_DIR}/xxxxx.lock. Since ${DL_DIR}
is read-only, it will cause IOError exception. Although
lockfile() can catch the exception, currently code simply
ignore all the exception and continue the loop. it make
sense if the exception is caused by locking contention,
but in the read-only $DL_DIR case, it cause endless waiting
unfortunately.
So this patch add read-only check for lockfile to avoid the
silent hang.
Fix [BUGID #606]
Signed-off-by: Yu Ke <ke.yu@intel.com>
Stupdi typo/thinko from me had depexp exiting once recipes had parsed
as I'd used a return the while loop where I'd meant a continue...
Signed-off-by: Joshua Lock <josh@linux.intel.com>
In previous exec() model, cooker is re-initialized from scratch with environmental
variable exported accordingly. Now in fork() model, environmental variables are
not exported again, and thus original method to export BB_TASKHASH doesn't apply
now which breaks all sstate packages. Now we can set data variable directly instead.
Signed-off-by: Kevin Tian <kevin.tian@intel.com>
If the checksum check failed, the .md5 stamp file would still have been created
meaning subsequent builds would proceed with the corrupt file. Reorder the calls
to avoid this. Also raise a specific error for the checksum not specified error
case.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This patch add the per-recipe SRC_URI checksum verification.
- SRC_URI format
The format of SRC_URI checksum follow OE definition:
1. SRC_URI has single src
SRC_URI = "http://some.domain/file.tar.gz"
SRC_URI[md5sum] = "xxxxxxxxxxxxxxx"
SRC_URI[sha256sum] = "xxxxxxxxxxxxxxxxxxxxxx"
2. SRC_URI has multiple src, every src need specify name
SRC_URI = "http://some.domain/file1.tar.gz;name=name1 \
http://some.domain/file2.tar.gz;name=name2 "
SRC_URI[name1.md5sum] = "xxxxxxxxxxxxxxx"
SRC_URI[name1.sha256sum] = "xxxxxxxxxxxxxxxxxxxxxx"
SRC_URI[name2.md5sum] = "xxxxxxxxxxxxxxx"
SRC_URI[name2.sha256sum] = "xxxxxxxxxxxxxxxxxxxxxx"
- SRC_URI checking invocation:
the checksum checking is invoked in do_fetch phase,
so it can be invoked manually by
# bitbake -f -c fetch <recipe_name>
if recipes has no SRC_URI checksum item, bitbake will show warning:
"
WARNING: Missing SRC_URI checksum for xxxx.tar.gz, consider to add
SRC_URI[md5sum] = "5c69f16d452b0bb3d44bc3c10556c072"
SRC_URI[sha256sum] = "f4e0ada8d4d516bbb8600a3ee7d9046c9c79e38cd781df9ffc46d8f16acd1768"
"
thus recipe author can add it to recpie file after SRC_URI
- control variable BB_STRICT_CHECKSUM
when SRC_URI checksum is missing, this variable decide pass or not
if BB_STRICT_CHECKSUM = "1", bitbake should fatal in this case, otherwise bitbake just pass
Signed-off-by: Yu Ke <ke.yu@intel.com>
sstate hash validation is done at initialization of RunQueueExecuteScenequeue.
However the index of 'valid' list returned from the validation doesn't
correspond to setscene task ID. It's just an intermediate namespace between
runqueue and sstate hash func. Use it as setscene task ID fully mess the flow.
Previously this doesn't cause trouble because all setscene tasks are passed. Commit
58396a5d24 add 'noexec' concept to setscene
tasks which grabs some tasks out of the list and thus trigger this problem
Without this fix there're ~50 recipes (gzip-native, glib, ...) rebuilt weirdly
with a minimal build, even though existing sstate packages could accelerate them.
there's another typo using wrong task ID in a debug message which further hide
this issue
Signed-off-by: Kevin Tian <kevin.tian@intel.com>
Add a parameter to getVarFlag() to auto-expand the value of the flag. This
makes getVarFlag() more consistent with getVar(), and allows expansion of
vardeps and vardepsexclude (which has been done in this commit).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Adds a vardepsexclude flag that can be used to exclude a dependency of
a variable (the opposite of vardeps). This will allow the exclusion of
variables from the hash generation much more selectively than blanket
whitelisting using BB_HASHBASE_WHITELIST.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Currently, anything whitelisted in the environment makes it into the worker
processes. This is undesireable and the worker environment should be as
clean as possible. This patch adapts bitbake sosme variables are loaded into
bitbake's datastore but not exported by default. Any variable can be exported
by setting its export flag.
Currently, this code only finalises the environment in he worker as doing so
in the server means variables are unavailable in the worker. If we switch
back to fork() calls instead of exec() this code will need revisting.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
idle command processing in each of the servers does not handle an explicit
None return value, which means the goggle UI ends up repeatedly adding
"Tasks Summary:" rows to the list.
This patch modifies BBCooker.buildTargets.buildTargetsIdle to return False
when BuildCompleted is fired, as is done in BBCooker.buildFile.buildFileIdle.
It may be that the correct way to fix this is to change the idle command
processing in the servers.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Construct a ProgressBar and pass it to the RunningBuild.handle_event() so
that goggle users are notified of metadata parsing progress.
UI's with status make users less nervous
Signed-off-by: Joshua Lock <josh@linux.intel.com>
It seems likely that the user would want to view the most recently emitted
messages so this patch sets the message dislay treeview to scroll to any
newly added rows.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Python 2.7's library changes some of xmlrpclib's internal implementation such
that interacting with a proxy to BitBakes SimpleXMLRPCServer would cause
BitBake to crash.
The issue was traced to changes in the xmlrpclib.Transport implementation and
Python bug #8194 (http://bugs.python.org/issue8194).
This patch introduces a workaround by create a subclass of
xmlrpclib.Transport, which overrides the offending methods with the Python
2.6.6 implementation copy and pasted from the Python 2.6.6 xmlrpclib, and
using this BBTransport implementation for both xmlrpclib.Server objects we
create.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
ProgressBar will be useful in other UI elements so make it it's own class.
Make ProgressBar a subclass of gtk.Dialog, rather than gtk.Window, so that we
can suggest the window manager parent the ProgressBar to the widget passed at
as parent.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Due to some recent change *somewhere* we need to explicitly look at the
name attribute on the instances class, rather than the name attribute of
the instance.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Move the test for ignored messages to the start of the message handling loop to
avoid doing work for messages which are only going to be ignored.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Some of the events we where trying to look for have had their class names
changed, fix these references.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
BB_TASKHASH_WHITELIST is expected to filter out native tasks from the
dependency list for target recipe's checksum. However current code
actually implements the opposite. All native sstate packages end up
to have empty task dependency while target sstate packages still have
native tasks counted into the checksum.
Signed-off-by: Kevin Tian <kevin.tian@intel.com>
Where a variable name consisted of an append/prepend combined with an override
and there was also an append/prepend to the variable, the override could be lost
if the override was not in OVERRIDES.
For example:
FOO = "A"
FOO_append = "B"
FOO_append_virtclass-native = "C"
could result in "AB" even though virtclass-native was in OVERRIDES.
With this patch applied, the result is "ABC" as would be expected.
The problem was the deletion of the _append/_prepend flag was happening
if *any* append/prepend was procesed, the result should really be that
it should contain any unprocessed append/prepend.
Kevin Tian deserves credit for looking into this and working out the
problem here.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Now 'runtaskdeps' is assigned before taskhash whitelist is filtered out, and thus
always contains original task dependencies. This causes problem to diagnose sstate
related problem. So fix it by appending filtered dep to 'runtaskdeps' in the fly.
Signed-off-by: Kevin Tian <kevin.tian@intel.com>
This means the noexec messages are only shown once as the stamp files are now
correctly created.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
two dep lists should be sorted before the comparison, or else it just
causes false-positive confusion.
Also fix 'taskdeps' to 'runtaskdeps' when printing out the difference.
Signed-off-by: Kevin Tian <kevin.tian@intel.com>
If sstate was used to accelerate a build, the pseudo directory might not have
been created leading to subsequent task failures.
Also, sstate packages were not being installed under pseudo context meaning
file permissions could have been lost.
Fix these problems by creating a FAKEROOTDIRS variable which bitbake ensures
exists before running tasks and running the appropriate setscene tasks under
fakeroot context.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
The git fetcher was failing to pull in new branches into a git
repository mirror tarball as the git fetch command being used didn't
add new remote branches.
This patch uses "git fetch --all" for fullclones to ensure any
new remote branches are cloned correctly.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
The more we try and patch up the fetcher code, the more things break. The
code blocks in question are practically unreadable and are full of corner
cases where fetching could fail. In summary the issues noticed included:
a) Always fetching strange broken urls from the premirror for "noclone"
git repositories
b) Not creating or rewriting .md5 stamp files inconsistently
c) Always fetching git source mirror tarballs from the premirror even
if they already exist but the checkout directory does now
d) Passing "None" values to os.access() and os.path.extsts() checks under
certain circumstances
e) Not using fetched git mirror tarballs if the preexist and always
try and fetch them.
This patch rewrites the sections of code in question to be simpler and
more readable, fixing the above problems and most likely other odd
corner cases.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Per function logging made it hard to track down what was happening so
switch to logging on a per task basis.
[BUGID #383]
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
We already export http,ftp,https proxy environment variables. Some environments,
GNOME for instance, place the socks proxy in ALL_PROXY and all_proxy. Export it
as well.
Signed-off-by: Darren Hart <dvhart@linux.intel.com>
We catch any exception raised by the fetchers go() method and attempt to work
around it by trying any (post) mirrors which are configured. However, should
the mirrors fail the exception is lost and the fetch is assumed to have
completed successfully.
Instead, save the exception and if the local file does not exist after trying
the mirrors re-raise the exception.
Fixes [BUGID #362]
Signed-off-by: Joshua Lock <josh@linux.intel.com>
[BUGID #375]
When configuring for the log file output, we generate a symlink to the
target filename. This link uses the same file naming, but without the
active pid, making it easier to see which log file is the last one
generated.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Pass task has informaiton to work processes, allowing full manipulation of
the hash data in the task context allowing checksums to be usable.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
With "no_proxy" exported to bitbake, both internal and externel file mirror can
be used. "https_proxy" enable fetching "https://" file through proxy.
Signed-off-by: Zhai Edwin <edwin.zhai@intel.com>
It was possible for bitbake-runtime to be run against a semi-installed
python-native resulting in tracebacks with ImportError's.
To prevent this we stash the initial PATH in the BBConfiguration when bitbake
is started and then set this in the env when launching bitbake-runtask through
subprocesses Popen() call.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Add a new method, try_premirror, to the Fetch object which checks to see
whether the file needs to be fetched from a premirror.
Override this in the Git fetcher to only require a pre-mirror fetch when the
clone directory does not exist.
Fixes [BUGID 290]
Signed-off-by: Joshua Lock <josh@linux.intel.com>
When setting localpath in the SRC_URI parameters the basename should be set
explicitly as the localpath() method is not called.
Fixes [BUGID #81]
Signed-off-by: Joshua Lock <josh@linux.intel.com>
If we fetch a tarball from a mirror it's entirely possible that the
mirror will have been from before the required tag/branch/etc was included
in the repository. To that end use forcefetch() as a way of testing whether
the repository is up to date and if not fetch updates.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
The git fetcher should force a fetch if the required tag is not present in
the local clone, or if the fullclone parameter is set.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
When pulling from a premirror we would prefer a local tarball even when the
caller had specified the forcefetch parameter.
Add an extra parameter 'force' to try_mirrors, defaulting to False. If set
the mirrors will be tested even if the file exists locally.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
For a given input to this code, the output doesn't change to implement a persistent
cache of the data to speed up parsing.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
These functions allow generation of dependency data between funcitons and
variables allowing moves to be made towards generating checksums and allowing
use of the dependency information in other parts of bitbake.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This commit is derived from Chris Larson's checksum work, turned into a
standalone piece of code for parsing python and shell functions.
The deindent code has been replaced with code to work around indentation
for speed. The original NodeVisitor in the ast was replaced with a faster
class walk call.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Add a new extend_deps function to more easily merge two dependency lists.
This avoids adding duplicates, unless the value of the dependency is
different.
Signed-off-by: Mark Hatle <mhatle@windriver.com>
It appears the timeout sometimes has no effect and we see database access failures. Combat
this by wrapping the execute function in all cases and retrying manually ourselves.
Thanks to Kevin Tian for help debugging this.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
bitbake: scenequeue: Skip setscene if the underlying task already ran
bitbake/setscene: Make sure uneeded dependencies are removed recursively
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
the 'fn' argument of them is not used, should be removed anytime:
sed -i -e
'/^def.*fn/s/,[[:space:]]*fn[[:space:]]*=[[:space:]]*None[[:space:]]*)/)/g'
lib/bb/msg.py
(Bitbake rev: 1cb72e371322c271ee7f2d008c6f7899fb38b4fd)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Searches the module (bb.runqueue) for any new style classes which are
instances of RunQueueScheduler, and uses the one whose 'name' attribute
matches the value of BB_SCHEDULER.
(Bitbake rev: 6497cedf9cfc03201250af816995dd2bd85c36ef)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
to make python3 happy
(Bitbake rev: c82926ccdd4ec4e3ad6e78a381dacb96adf9b409)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Likely cause is, of course, typo in the pattern or incorrect BBFILES, so we
should warn the user about this.
(Bitbake rev: b781317b5006bc047a59e7fa3c93344115e78ccb)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This failure is one the *user* needs to see, as it may have to do with their
setup, but where they don't need to see a traceback and all, since they're not
developers, so I think SystemExit is appropriate here.
(Bitbake rev: e3cc9bc7b6ef1a2eca78235790a051329bceb9a4)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This is necessary, as there has not yet been a release with the bb.utils
versions in place. We can't show them a deprecation warning when they can't
safely change it yet. Show a PendingDeprecationWarning until 1.10 releases
and OE requires it, then we can move to DeprecationWarning.
(Bitbake rev: 43700fa1ca1d2635de4ed46110057cb164ee82d1)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Fixes proxy support to honor standard proxy environment variables.
* Quote environment variables
(Bitbake rev: f84f382f340d6db15b9e5afb8c7c93969249a958)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
time.strftime() defaults to using time.localtime(), use that instead
of spawning "sh -c 'date +<FMT>'"
(Bitbake rev: ec1f2773b4811b3beb52710120bb5a9b215c174e)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
previously we called chdir() twice for every target, this patch reduces
the amount of chdir() calls via openembedded master from some 16000 to
4.
(Bitbake rev: fa45f5625e13a82bec70d5f10815f52fbe705166)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Store database connection to persistent database in fetcher.
(Bitbake rev: 8a6876752b90efd81d92f0947bfc9527d8260969)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
to make updating depends easier/more intuitive/eventually faster
(Bitbake rev: f7c69462b8ba726861898817cc5b13174c78e35a)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
(Bitbake rev: c0cf85beda4cf8748fd270c037442cde7b98146b)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
(Bitbake rev: 09fe0d403c75229931c10eabc7eecc0dc60058fe)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
The root cause is, I was testing the use of renameVar() from finalize, in
order to get flags copied over when an override is applied, but renameVar
removes the original, whereas the old code did not do so. Going back to the
old method, will revisit the override/flags later on.
(Bitbake rev: 2f7c498abcf675e5b8de197d8056a0581670c2bd)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This ensures the check is run for anyone using the package, including
setup.py, though it's sad that it's necessary, since all it really wants is
__version__.
Ideally, we'd avoid the version check entirely in favor of checking for
functionality, but that's rather difficult with language constructs like
context managers, so this'll do for now.
(Bitbake rev: f936f1c0edf70860ea0f39ec1a500ba43b56f289)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
See the comment in the code for details.
(Bitbake rev: 849dbd63244cbc4eaca0f1beedbb67baca024629)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This way we can fully utilize bblayers, you can do everything in bblayers.conf
and avoid setting any environment variables at all.
(Bitbake rev: 5def1c8c31432968349f9b29d6333d7962260a8b)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
types.IntType -> int
types.StringType -> basestring
...
Also moves our ImmutableTypes tuple into our own namespace.
(Bitbake rev: 83674a3a5564ecb1f9d2c9b2d5b1eeb3c31272ab)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
As an example, this displays a deprecation warning for the use of
"bb.encodeurl" when you should be using "bb.fetch.encodeurl". It includes a
convenience function for this purpose. It should be of use when moving
objects between modules permanently, changing the API the user sees.
(Bitbake rev: 78f56049ba863b2e585b89db12b32697eb879bbc)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
We should add back in some proper unit testing. Doctest is rather limited,
and we haven't actually made use of it since the original implementation of
the datastore when the project started, as far as I'm aware.
(Bitbake rev: 3a11c2807972bbbddffde2fa67fc380d159da467)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This lets you do -l Default on the commandline to bump the debug level of
messages with no domain, without bumping the level of everything else.
(Bitbake rev: bef33dfc36713d8c93e38e63ddac2e4f9858e787)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Also fixes some bb.msg references from within bb.msg.
(Bitbake rev: db95af590f742c8186e84046ad9704fae1733720)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Apparently the finalise spelling is becoming less common in British English.
(Bitbake rev: 47449b2fc433e5725839ca4f7e9bca931a475838)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
We should add back in some proper unit testing. Doctest is rather limited,
and we haven't actually made use of it since the original implementation of
the datastore when the project started, as far as I'm aware.
(Bitbake rev: 3a11c2807972bbbddffde2fa67fc380d159da467)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
When BBCLASSEXTEND is set, '-b' builds usually failed with messages
like
| ERROR: Parsing error data_fn virtual:native:<recipe>.bb and fn
<recipe>.bb don't match
|
| File ".../bb/providers.py", line 47, in sortPriorities
| priority = dataCache.bbfile_priority[f]
| KeyError: 'virtual:native:<recipe>.bb'
This patch fixes it and allows to specify the alternative class in a
way like
| ./bitbake -b virtual:native:<recipe>.bb
This patch was written to be so minimal as possible; variables should be
probably renamed to reflect their new meaning.
(Bitbake rev: f1c7fe9fc12161ceb3fe201cde370b929b208729)
Signed-off-by: Enrico Scholz <enrico.scholz@informatik.tu-chemnitz.de>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This is just for compatibility. We may drop it in the future, or rewrite it,
as it's not particularly pythonic.
(Bitbake rev: c4e31d7fe1d15a1e3ef2a453e7c7812d403d22cb)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Cache only uses the config metadata from the cooker, no need for the cooker itself.
(Bitbake rev: bf58d43444642d31293a341ce72292003ca5c162)
Signed-off-by: Chris Larson <clarson@kergoth.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
The methodpool, ${@} expansions, anonymous python functions, event handlers
now all run with the same global context, ensuring a consistent environment
for them. Added a bb.utils.better_eval function which does an eval() with the
same globals as better_exec.
(Bitbake rev: 424d7e267b009cc19b8503eadab782736d9597d0)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
- Use a single dictionary for the context, both global & local, since for some
reason it chokes wanting a global "d" rather than a local in the metadata.
- First compile the string into a code object before running eval, so we can
include the variable name in an evaluation error.
(Bitbake rev: 49534d928a37e0804ca84eed186cd22363023b2e)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
In order to move the environment cleaning, which wants to log a
message, before cooker creation, the logging facility initialization
needs to happen earlier. So, it is now in init_logger in utils.py and
called from bitbake before the creation of the data object.
It also seems more correct to initialize a global facility like this
from a more global context than the creation of an object, of which
there could theoretically be many.
Signed-off-by: Jeff Dike <jdike@linux.intel.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Without this patch decoding a url of this kind file://dir/filename gives
path=/filename host=dir.
With the patch it decodes as path=/dir/filename host=""
Probably nobody stumbled on this issue yet because nobody used
file:// urls with directory names in the path.
Signed-off-by: Nitin A Kamble <nitin.a.kamble@intel.com>
The following utility functions were removed. Note that in this particular
case we're bypassing a proper deprecation process, as these functions are
clearly not utilized, are obvious remnants of old ways of doing things, and
some of which do not even function properly.
- tokenize
- evaluate
- flatten
- relparse
- ververify
- isjustname
- isspecific
- catpkgsplit
- pkgsplit
- pkgcmp
- dep_parenreduce
- dep_opconvert
(Bitbake rev: 62983ad9b151ee8d51e8cf9a31c736c7813edf16)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This implements a feature similar to BBCLASSEXTEND, but for generating
multiple versions of a given recipe. For example: BBVERSIONS = "1.0 2.0 git".
In addition to the above, one can utilize [a-b] style patterns, and can have a
:<basever> postfix, which allows you to essentially name the range of
versions. Both the current version and the basever end up in OVERRIDES, and
the basever gets placed into the BPV variable. The default BPV, if none is
specified, is the original PV of the recipe, before bbversions processing.
In this way, you can do things like:
BBVERSIONS = "1.0.[0-6]:1.0.0+
1.0.[7-9]:1.0.7+"
SRC_URI_append_1.0.7+ = "file://some_extra_patch.patch;patch=1"
Or you can create a recipe per range, and name the recipe file as such: nano_1.0.7+.bb.
(Bitbake rev: 4ee9a56e16f1eb3c1649eaa3127b09ab0e93d1ec)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
When not expanding PREMIRRORS, the functions fails/does not work correctly
when PREMIRRORS is not a plain string (e.g. contains ${...} or a ${@...}
statements).
(Bitbake rev: d612d22b073f68b8cf1bb7344e0487820040d80d)
Signed-off-by: Enrico Scholz <enrico.scholz@informatik.tu-chemnitz.de>
Signed-off-by: Chris Larson <clarson@kergoth.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
(Bitbake rev: b2486ec57c6a7adf09d0960fdf6727881b324d2f)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Chris Larson <clarson@kergoth.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This prevents a misleading backtrace:
ERROR: no files to build.
Command execution failed: Traceback (most recent call last):
File ".../bitbake/build/lib/bb/command.py", line 83, in runAsyncCommand
self.cooker.updateCache()
File ".../bitbake/build/lib/bb/cooker.py", line 779, in updateCache
if not self.parser.parse_next():
File ".../bitbake/build/lib/bb/cooker.py", line 969, in parse_next
cooker.bb_cache.sync()
UnboundLocalError: local variable 'cooker' referenced before assignment
(Bitbake rev: 060ef3d957615b7eb1209dc0d01ebeb53f8c4edc)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Chris Larson <clarson@kergoth.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
??= is a lazy, conditional assignment. Whereas a ?= immediately assigns to
the variable if the variable has not yet been set, ??= does not apply the
default assignment until the end of the parse. As a result, the final ??= for
a given variable is used, as opposed to the first as in ?=.
Note that the initial implementation relies upon finalise() to apply the
defaults, so a "bitbake -e" without specifying a recipe will not show the
defaults as set by ??=. Moving application of the default into getVar adds
too large a performance hit. We may want to revisit this later.
(Bitbake rev: 74f50fbca194c9c72bd2a540f4b9de458cb08e2d)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
dict objects provide an __iter__ method for the iteration which gives you the
keys, so calling keys directly is unnecessary, and isn't really a best
practice. The only time you really need to call the keys is if there's a
danger of the dict changing out from underneith you, either due to external
forces or due to modification of the iterable in the loop. Iterations over
os.environ are apparently subject to such changes, so they must continue to
use keys().
As an aside, also switches a couple spots to using sorted() rather than
creating a temporary list with keys() and sorting that.
(Bitbake rev: 5b6ccb16c6e71e23dac6920cd2df994d67c2587b)
Signed-off-by: Chris Larson <clarson@mvista.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
..to make copy and paste of the logfile easier.
(Bitbake rev: 446cc0cebd4daff7f849717f4cb89ac1b4c6b755)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Chris Larson <clarson@kergoth.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
(Bitbake rev: f68406e864c9837feb56cbec993b620481445cc2)
Signed-off-by: Tom Rini <tom_rini@mentor.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This ensures that an anonymous python function is able to manipulate the
BBCLASSEXTEND contents, and, therefore, amend.inc files are able to add to it.
(Bitbake rev: c7d038d404afaf4ce3735af5134163759da6f6ef)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Better to error as early as possible rather than experience strange behavior
resulting from the use of the largely useless stock bitbake.conf/base.bbclass.
(Bitbake rev: 641e6cf3ec3ab4d26929cf4d2a3704ff07eed4d6)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
The Git fetcher currently hardwires the git command to "git". Allow the
path and any additional wrappers to the Git command to be provided via
FETCHCMD functionality, as with some of the other fetchers.
If FETCHCMD_git is not define in bitbake.conf, the fetcher defaults to "git".
(Bitbake rev: f3afb79ecac30d973a3c62ff6baf28d8b7388a24)
Signed-off-by: Malcolm Crossley <malcolm.crossley@ge.com>
Signed-off-by: Martyn Welch <martyn.welch@ge.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This breaks preferred providers functionality
This reverts commit ee9afccf33b220a21b74fab279925eeb4771249b.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Do not attempt to open the file in the resolve_file method
(a lot like bb.which... maybe bb.which can be used). This way
we don't need to open/close a file which we have already parsed.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Have a growing dict with .inc and .bbclass'es. This avoids to reparse
files we have already seen.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Our parser is shit but instead to replace it now we will see
how long we can drive the wave by caching parsed files. This
will not go through the feeder again but we can just reevaluate
the StatementGroup.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Evaluate the statements after having parsed one file. This is
referred to as "entwirren" and we can remove the direct evaluation
and postpone a bit, in the future we can use a cached copy instead
of parsing the original.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
First set of ConfHandling with AST nodes. The include can
use a speed up and things might need to be migrated... into
this class.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
When parsing we will collect a number of statements
that can be evaluated...The plan is to be evaluate
things twice (old+new) and then compare the result,
it should be the same.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* lineno and fn are needed in handleMethod
to restore the functionality as it was before.
(Bitbake rev: ac6792045959cfee56279c1c4597521e990848e7)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Chris Larson <clarson@kergoth.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
We want to convert this into a proper AST. So move all
such operations to methods... Later change them to generate
a node... and create that node from here.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
With obtain it was possible to use an existing fetcher to
download a bb or config file. In practive no one has used it
and it was likely broken in regard to depends_cache... Remove
it for now, simplfiy the code.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
checkstatus() is used to ensure we can fetch a copy of each file, so it makes
sense to also test PREMIRRORS and MIRRORS in the method.
This patch adds calls to try_mirrors() to the Fetch.checkstatus() method and
changes the try_mirrors() method to take a check argument, which is False by
default. When check is True try_mirrors() will call a fetchers checkstatus()
with the replaced uri.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
The logic in try_mirrors() was buggy such that only the first entry in the
mirrors list was tried.
This patch fixes this. Thanks to Richard for the review.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
When we are using PREMIRRORS it's possible a mirror in the local namespace
(some filesystem path, i.e. an NFS share) provides read-only files.
This is a perfectly valid scenario so this patch fixes bitbake so that for
such a scenario locapath is set to the files path rather than some child
of DL_DIR.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Modify the try_mirrors() function to return the localpath of the fetched file
and update the data dictionary to reflect this.
Secondly the metadata files, lock and md5, should always be stored relative to
the ${DL_DIR} as it is possible that the localpath is a read-only directory,
for example in the scenario where there is a read-only file:// mirror.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
This patch serves two purposes. Firstly it unifies the concept of mirrors into
PREMIRRORS and MIRRORS. PREMIRRORS are tried before the SRC_URI defined in the
recipe whereas MIRRORS are tried only if that fails.
The tarball stash was conceptually inline with a PREMIRROR only with special
handling within the wget fetcher and therefore only worked with certain
fetch types.
Secondly the patch removes the need for individual fetch implementations to
worry about mirror handling.
With this patch, the base fetch implementation will first try to use a
PREMIRROR to fetch the desired object, if this fails the native fetch method
for the object will be tried and if this fails will try to fetch a copy from
one of the MIRRORS.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
If there was a variable such as:
X_${Y}_append = "Z"
The "Z" would be lost if X_${Y} was unset. This was due to a bug in the renameVar
function used by expandKeys().
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Make the test functionality work
* Optimise BBPATH handling when changing directory
* Optimise file globing for BBFILES
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Refactor the environment variable handling so that it more closes matches the
changes that have been made in bitbake-dev.
git-svn-id: https://svn.o-hand.com/repos/poky/trunk@5527 311d38ba-8fff-0310-9ca6-ca027cbcb966
The Mercurial fetcher right now will fail when used to incrementally
fetch an update to a local clone of a repository already fetched at
some prior revision. The culprit is the sequence:
hg pull -r <rev>
hg update -C <rev>
A subtlety in the way that Mercurial stores its tags (in a normally
version-controlled file called .hgtags) has the side-effect that a
repository fetched at a tag "foo" will not actually contain a
new-enough copy of the .hgtags file to be self-aware of the foo tag's
existence.
The solution is just to get all the changesets in the repository on
incremental upgrades, so that the following "hg update" will be able
to resolve the tag.
git-svn-id: https://svn.o-hand.com/repos/poky/trunk@5170 311d38ba-8fff-0310-9ca6-ca027cbcb966
* File licence headers were sanitised causing most of the diff.
* cooker.py was created from bin/bitbake.
* cvs fetcher port option was added
* The -f force option was fixed to work correctly
* Multiple entries in rrecrdeps are now handled correctly
(allows adding do_deploy to image depends)
git-svn-id: https://svn.o-hand.com/repos/poky/trunk@1129 311d38ba-8fff-0310-9ca6-ca027cbcb966
Chris Larson
Chris Larson
Yu Ke
Joshua Lock
Richard Purdie
Kevin Tian
Paul Eggleton
Darren Hart
Mark Hatle
Zhai Edwin
Mark Hatle
Bernhard Reutner-Fischer
Ryan Phillips
Khem Raj
Enrico Scholz
Jeff Dike
Nitin A Kamble
Chris Larson
Tom Rini
Martyn Welch
Holger Freyther
Martin Jansa
Richard Purdie
Mike Turquette
Robert Bragg
Richard Purdie
Richard Purdie
Robert Bradford
Joshua Lock
Marcin Juszkiewicz
Ross Burton
Chris Larson
Chris Lord