sstate hash validation is done at initialization of RunQueueExecuteScenequeue.
However the index of 'valid' list returned from the validation doesn't
correspond to setscene task ID. It's just an intermediate namespace between
runqueue and sstate hash func. Use it as setscene task ID fully mess the flow.
Previously this doesn't cause trouble because all setscene tasks are passed. Commit
58396a5d24 add 'noexec' concept to setscene
tasks which grabs some tasks out of the list and thus trigger this problem
Without this fix there're ~50 recipes (gzip-native, glib, ...) rebuilt weirdly
with a minimal build, even though existing sstate packages could accelerate them.
there's another typo using wrong task ID in a debug message which further hide
this issue
Signed-off-by: Kevin Tian <kevin.tian@intel.com>
Add a parameter to getVarFlag() to auto-expand the value of the flag. This
makes getVarFlag() more consistent with getVar(), and allows expansion of
vardeps and vardepsexclude (which has been done in this commit).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Adds a vardepsexclude flag that can be used to exclude a dependency of
a variable (the opposite of vardeps). This will allow the exclusion of
variables from the hash generation much more selectively than blanket
whitelisting using BB_HASHBASE_WHITELIST.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Currently, anything whitelisted in the environment makes it into the worker
processes. This is undesireable and the worker environment should be as
clean as possible. This patch adapts bitbake sosme variables are loaded into
bitbake's datastore but not exported by default. Any variable can be exported
by setting its export flag.
Currently, this code only finalises the environment in he worker as doing so
in the server means variables are unavailable in the worker. If we switch
back to fork() calls instead of exec() this code will need revisting.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
idle command processing in each of the servers does not handle an explicit
None return value, which means the goggle UI ends up repeatedly adding
"Tasks Summary:" rows to the list.
This patch modifies BBCooker.buildTargets.buildTargetsIdle to return False
when BuildCompleted is fired, as is done in BBCooker.buildFile.buildFileIdle.
It may be that the correct way to fix this is to change the idle command
processing in the servers.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Construct a ProgressBar and pass it to the RunningBuild.handle_event() so
that goggle users are notified of metadata parsing progress.
UI's with status make users less nervous
Signed-off-by: Joshua Lock <josh@linux.intel.com>
It seems likely that the user would want to view the most recently emitted
messages so this patch sets the message dislay treeview to scroll to any
newly added rows.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Python 2.7's library changes some of xmlrpclib's internal implementation such
that interacting with a proxy to BitBakes SimpleXMLRPCServer would cause
BitBake to crash.
The issue was traced to changes in the xmlrpclib.Transport implementation and
Python bug #8194 (http://bugs.python.org/issue8194).
This patch introduces a workaround by create a subclass of
xmlrpclib.Transport, which overrides the offending methods with the Python
2.6.6 implementation copy and pasted from the Python 2.6.6 xmlrpclib, and
using this BBTransport implementation for both xmlrpclib.Server objects we
create.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
ProgressBar will be useful in other UI elements so make it it's own class.
Make ProgressBar a subclass of gtk.Dialog, rather than gtk.Window, so that we
can suggest the window manager parent the ProgressBar to the widget passed at
as parent.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Due to some recent change *somewhere* we need to explicitly look at the
name attribute on the instances class, rather than the name attribute of
the instance.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Move the test for ignored messages to the start of the message handling loop to
avoid doing work for messages which are only going to be ignored.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Some of the events we where trying to look for have had their class names
changed, fix these references.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
BB_TASKHASH_WHITELIST is expected to filter out native tasks from the
dependency list for target recipe's checksum. However current code
actually implements the opposite. All native sstate packages end up
to have empty task dependency while target sstate packages still have
native tasks counted into the checksum.
Signed-off-by: Kevin Tian <kevin.tian@intel.com>
Where a variable name consisted of an append/prepend combined with an override
and there was also an append/prepend to the variable, the override could be lost
if the override was not in OVERRIDES.
For example:
FOO = "A"
FOO_append = "B"
FOO_append_virtclass-native = "C"
could result in "AB" even though virtclass-native was in OVERRIDES.
With this patch applied, the result is "ABC" as would be expected.
The problem was the deletion of the _append/_prepend flag was happening
if *any* append/prepend was procesed, the result should really be that
it should contain any unprocessed append/prepend.
Kevin Tian deserves credit for looking into this and working out the
problem here.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Now 'runtaskdeps' is assigned before taskhash whitelist is filtered out, and thus
always contains original task dependencies. This causes problem to diagnose sstate
related problem. So fix it by appending filtered dep to 'runtaskdeps' in the fly.
Signed-off-by: Kevin Tian <kevin.tian@intel.com>
This means the noexec messages are only shown once as the stamp files are now
correctly created.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
two dep lists should be sorted before the comparison, or else it just
causes false-positive confusion.
Also fix 'taskdeps' to 'runtaskdeps' when printing out the difference.
Signed-off-by: Kevin Tian <kevin.tian@intel.com>
If sstate was used to accelerate a build, the pseudo directory might not have
been created leading to subsequent task failures.
Also, sstate packages were not being installed under pseudo context meaning
file permissions could have been lost.
Fix these problems by creating a FAKEROOTDIRS variable which bitbake ensures
exists before running tasks and running the appropriate setscene tasks under
fakeroot context.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
The git fetcher was failing to pull in new branches into a git
repository mirror tarball as the git fetch command being used didn't
add new remote branches.
This patch uses "git fetch --all" for fullclones to ensure any
new remote branches are cloned correctly.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
The more we try and patch up the fetcher code, the more things break. The
code blocks in question are practically unreadable and are full of corner
cases where fetching could fail. In summary the issues noticed included:
a) Always fetching strange broken urls from the premirror for "noclone"
git repositories
b) Not creating or rewriting .md5 stamp files inconsistently
c) Always fetching git source mirror tarballs from the premirror even
if they already exist but the checkout directory does now
d) Passing "None" values to os.access() and os.path.extsts() checks under
certain circumstances
e) Not using fetched git mirror tarballs if the preexist and always
try and fetch them.
This patch rewrites the sections of code in question to be simpler and
more readable, fixing the above problems and most likely other odd
corner cases.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Per function logging made it hard to track down what was happening so
switch to logging on a per task basis.
[BUGID #383]
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
We already export http,ftp,https proxy environment variables. Some environments,
GNOME for instance, place the socks proxy in ALL_PROXY and all_proxy. Export it
as well.
Signed-off-by: Darren Hart <dvhart@linux.intel.com>
We catch any exception raised by the fetchers go() method and attempt to work
around it by trying any (post) mirrors which are configured. However, should
the mirrors fail the exception is lost and the fetch is assumed to have
completed successfully.
Instead, save the exception and if the local file does not exist after trying
the mirrors re-raise the exception.
Fixes [BUGID #362]
Signed-off-by: Joshua Lock <josh@linux.intel.com>
[BUGID #375]
When configuring for the log file output, we generate a symlink to the
target filename. This link uses the same file naming, but without the
active pid, making it easier to see which log file is the last one
generated.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Pass task has informaiton to work processes, allowing full manipulation of
the hash data in the task context allowing checksums to be usable.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
With "no_proxy" exported to bitbake, both internal and externel file mirror can
be used. "https_proxy" enable fetching "https://" file through proxy.
Signed-off-by: Zhai Edwin <edwin.zhai@intel.com>
It was possible for bitbake-runtime to be run against a semi-installed
python-native resulting in tracebacks with ImportError's.
To prevent this we stash the initial PATH in the BBConfiguration when bitbake
is started and then set this in the env when launching bitbake-runtask through
subprocesses Popen() call.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Add a new method, try_premirror, to the Fetch object which checks to see
whether the file needs to be fetched from a premirror.
Override this in the Git fetcher to only require a pre-mirror fetch when the
clone directory does not exist.
Fixes [BUGID 290]
Signed-off-by: Joshua Lock <josh@linux.intel.com>
When setting localpath in the SRC_URI parameters the basename should be set
explicitly as the localpath() method is not called.
Fixes [BUGID #81]
Signed-off-by: Joshua Lock <josh@linux.intel.com>
If we fetch a tarball from a mirror it's entirely possible that the
mirror will have been from before the required tag/branch/etc was included
in the repository. To that end use forcefetch() as a way of testing whether
the repository is up to date and if not fetch updates.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
The git fetcher should force a fetch if the required tag is not present in
the local clone, or if the fullclone parameter is set.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
When pulling from a premirror we would prefer a local tarball even when the
caller had specified the forcefetch parameter.
Add an extra parameter 'force' to try_mirrors, defaulting to False. If set
the mirrors will be tested even if the file exists locally.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
For a given input to this code, the output doesn't change to implement a persistent
cache of the data to speed up parsing.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
These functions allow generation of dependency data between funcitons and
variables allowing moves to be made towards generating checksums and allowing
use of the dependency information in other parts of bitbake.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This commit is derived from Chris Larson's checksum work, turned into a
standalone piece of code for parsing python and shell functions.
The deindent code has been replaced with code to work around indentation
for speed. The original NodeVisitor in the ast was replaced with a faster
class walk call.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Add a new extend_deps function to more easily merge two dependency lists.
This avoids adding duplicates, unless the value of the dependency is
different.
Signed-off-by: Mark Hatle <mhatle@windriver.com>
It appears the timeout sometimes has no effect and we see database access failures. Combat
this by wrapping the execute function in all cases and retrying manually ourselves.
Thanks to Kevin Tian for help debugging this.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
bitbake: scenequeue: Skip setscene if the underlying task already ran
bitbake/setscene: Make sure uneeded dependencies are removed recursively
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
the 'fn' argument of them is not used, should be removed anytime:
sed -i -e
'/^def.*fn/s/,[[:space:]]*fn[[:space:]]*=[[:space:]]*None[[:space:]]*)/)/g'
lib/bb/msg.py
(Bitbake rev: 1cb72e371322c271ee7f2d008c6f7899fb38b4fd)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Searches the module (bb.runqueue) for any new style classes which are
instances of RunQueueScheduler, and uses the one whose 'name' attribute
matches the value of BB_SCHEDULER.
(Bitbake rev: 6497cedf9cfc03201250af816995dd2bd85c36ef)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
to make python3 happy
(Bitbake rev: c82926ccdd4ec4e3ad6e78a381dacb96adf9b409)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Likely cause is, of course, typo in the pattern or incorrect BBFILES, so we
should warn the user about this.
(Bitbake rev: b781317b5006bc047a59e7fa3c93344115e78ccb)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This failure is one the *user* needs to see, as it may have to do with their
setup, but where they don't need to see a traceback and all, since they're not
developers, so I think SystemExit is appropriate here.
(Bitbake rev: e3cc9bc7b6ef1a2eca78235790a051329bceb9a4)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This is necessary, as there has not yet been a release with the bb.utils
versions in place. We can't show them a deprecation warning when they can't
safely change it yet. Show a PendingDeprecationWarning until 1.10 releases
and OE requires it, then we can move to DeprecationWarning.
(Bitbake rev: 43700fa1ca1d2635de4ed46110057cb164ee82d1)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Fixes proxy support to honor standard proxy environment variables.
* Quote environment variables
(Bitbake rev: f84f382f340d6db15b9e5afb8c7c93969249a958)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
time.strftime() defaults to using time.localtime(), use that instead
of spawning "sh -c 'date +<FMT>'"
(Bitbake rev: ec1f2773b4811b3beb52710120bb5a9b215c174e)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
previously we called chdir() twice for every target, this patch reduces
the amount of chdir() calls via openembedded master from some 16000 to
4.
(Bitbake rev: fa45f5625e13a82bec70d5f10815f52fbe705166)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Store database connection to persistent database in fetcher.
(Bitbake rev: 8a6876752b90efd81d92f0947bfc9527d8260969)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
to make updating depends easier/more intuitive/eventually faster
(Bitbake rev: f7c69462b8ba726861898817cc5b13174c78e35a)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
(Bitbake rev: c0cf85beda4cf8748fd270c037442cde7b98146b)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
(Bitbake rev: 09fe0d403c75229931c10eabc7eecc0dc60058fe)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
The root cause is, I was testing the use of renameVar() from finalize, in
order to get flags copied over when an override is applied, but renameVar
removes the original, whereas the old code did not do so. Going back to the
old method, will revisit the override/flags later on.
(Bitbake rev: 2f7c498abcf675e5b8de197d8056a0581670c2bd)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This ensures the check is run for anyone using the package, including
setup.py, though it's sad that it's necessary, since all it really wants is
__version__.
Ideally, we'd avoid the version check entirely in favor of checking for
functionality, but that's rather difficult with language constructs like
context managers, so this'll do for now.
(Bitbake rev: f936f1c0edf70860ea0f39ec1a500ba43b56f289)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
See the comment in the code for details.
(Bitbake rev: 849dbd63244cbc4eaca0f1beedbb67baca024629)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This way we can fully utilize bblayers, you can do everything in bblayers.conf
and avoid setting any environment variables at all.
(Bitbake rev: 5def1c8c31432968349f9b29d6333d7962260a8b)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
types.IntType -> int
types.StringType -> basestring
...
Also moves our ImmutableTypes tuple into our own namespace.
(Bitbake rev: 83674a3a5564ecb1f9d2c9b2d5b1eeb3c31272ab)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
As an example, this displays a deprecation warning for the use of
"bb.encodeurl" when you should be using "bb.fetch.encodeurl". It includes a
convenience function for this purpose. It should be of use when moving
objects between modules permanently, changing the API the user sees.
(Bitbake rev: 78f56049ba863b2e585b89db12b32697eb879bbc)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
