* without this fix, we get :
updating working directory
74 files updated, 0 files merged, 0 files removed, 0 files unresolved
abort: There is no Mercurial repository here (.hg not found)!
(Bitbake rev: 75ea005ac8fc05b2b3afca803d77a6b5f558efee)
Signed-off-by: Eric Bénard <eric@eukrea.com>
Tested-by: Frans Meulenbroeks <fransmeulenbroeks@gmail.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This was inadvertantly removed when trying to reduce the amount of duplicated
information the user sees when a failure occurs.
(Bitbake rev: 850d6158ea9daa58e896fd6b258d586df797dcf4)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Ensure it raises KeyError for a missing key, this is required to use this as a
mapping in various places, e.g. as locals in an eval.
(Bitbake rev: 8d661ce0c303e8d69f17c1d095545d5ed086d1d5)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This option will exclude the SCM metadata from tar files.
Tested with gcc where svn tar which used to be 156M for gcc 4.5
is now 77M
(Bitbake rev: f264cb6d43472525ad787b0887764ea696ec52ba)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
- Queue up any events fired to the UI before the UI exists
- At exit, check if UIs exist, and if not, flush the queue of LogRecords to
the console directly.
- When establishing a connection from the UI to the server, flush the queue of
events to the queue in the server connection, so the UI will receive them
when it begins its event loop.
(Bitbake rev: 73488aeb317ed306f2ecf99cc9d3708526a5933c)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
- Don't store key/value pairs when the value is None
- Delete the depends_cache when we're done with it
This reduces the memory usage after sync on initial parse by roughly 11.5% on
this machine.
(Bitbake rev: c7eb4c989459d182fdf9c81a627d32b7ef11626b)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
A SystemExit from a python function wasn't being raised as a FuncFailed, which
resulted in it not being caught by the exception handlers in the runqueue for
the worker process, which resulted in a SystemExit exit, rather than os._exit,
which causes all manner of problems when used in a forked process. This fixes
it by ensuring we raise a FuncFailed when seeing exceptions which aren't
instances of Exception.
(Bitbake rev: dafe92fe9f387450d9f9e9ff41c99388998b7495)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Per the python documentation, os.waitpid returns the exitcode shifted up by 8
bits, and we weren't compensating, resulting in a display of 'failed with 256'
when a worker process exits with a code of 1.
(Bitbake rev: 90c2b6cb24dc9c82f0a9aa9d23f2d1ed2e6ff301)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
- Drop EventException
- Use FuncFailed as the primary function failure exception, using TaskFailed
for the event (leaving it up to the process running exec_{func,task} to
display the more detailed information available in the exception).
- Switch InvalidTask to an exception rather than an event, as that's a
critical issue.
- Reduce the number of messages shown to the user when a task fails -- they
don't need to be told it fails 12 times. Work remains in this area though.
(Bitbake rev: 06b742aae2b8013cbb269cc30554cff89e3a5667)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
We use a custom Logger subclass for our loggers
This logger provides:
- 'debug' method which accepts a debug level
- 'plain' method which bypasses log formatting
- 'verbose' method which is more detail than info, but less than debug
(Bitbake rev: 3b2c1fe5ca56daebb24073a9dd45723d3efd2a8d)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This kills firing of Msg* events in favor of just passing along LogRecord
objects. These objects hold more than just level and message, but can also
have exception information, so the UI can decide what to do with that.
As an aside, when using the 'none' server, this results in the log messages in
the server being displayed directly via the logging module and the UI's
handler, rather than going through the server's event queue. As a result of
doing it this way, we have to override the event handlers of the base logger
when spawning a worker process, to ensure they log via events rather than
directly.
(Bitbake rev: c23c015cf8af1868faf293b19b80a5faf7e736a5)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
(Bitbake rev: f7c181a0f6ab0b4d33bf80a0e24a788de441f82b)
Signed-off-by: C Michael Sundius <msundius@sundius.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Provide __len__, __iter__, and the getitem/setitem/delitem methods, and its
mixed in versions of keys(), values(), items(), etc will automatically behave,
making the DataSmart act more like a real mapping.
(Bitbake rev: 89b5351c656d263b0ce513cee043bc046d20a01e)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
It needs to be a generator, so scheduler subclasses have the option to skip
buildable tasks and return a later one.
(Bitbake rev: a8c61e41bc6277222e4cde667ad0b24bd1597aa0)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
If you create a runqueue scheduler class in a python module, available in the
usual python search path, you can now make it available to bitbake via the
BB_SCHEDULERS variable, and the user can then select it as they select any
other scheduler.
Example usage:
In a test.py I placed appropriately:
import bb.runqueue
class TestScheduler(bb.runqueue.RunQueueScheduler):
name = "myscheduler"
In local.conf, to make it available and select it:
BB_SCHEDULERS = "test.TestScheduler"
BB_SCHEDULER = "myscheduler"
(Bitbake rev: 4dd38d5cfb80f9bb72bc41a629c3320b38f7314d)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
SIGINT should be from the user, not a script. It also doesn't work as
reliably to shut down processes, as it's not always interpreted as a
termination request. In addition, it causes KeyboardInterrupt exceptions in
the worker processes, which can interfere with our exception handling.
(Bitbake rev: e5f6e0e9de4c6d1dfdd269d2bf7f83c00c415a27)
Signed-off-by: Chris Larson <clarson@kergoth.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Bug 606 report that if $DL_DIR is read-only, do_fetch will
simply hang without any error message.
The root cause is that: bb.fetch.go()->bb.utils.lockfile()
will try to lock file ${DL_DIR}/xxxxx.lock. Since ${DL_DIR}
is read-only, it will cause IOError exception. Although
lockfile() can catch the exception, currently code simply
ignore all the exception and continue the loop. it make
sense if the exception is caused by locking contention,
but in the read-only $DL_DIR case, it cause endless waiting
unfortunately.
So this patch add read-only check for lockfile to avoid the
silent hang.
Fix [BUGID #606]
Signed-off-by: Yu Ke <ke.yu@intel.com>
Stupdi typo/thinko from me had depexp exiting once recipes had parsed
as I'd used a return the while loop where I'd meant a continue...
Signed-off-by: Joshua Lock <josh@linux.intel.com>
In previous exec() model, cooker is re-initialized from scratch with environmental
variable exported accordingly. Now in fork() model, environmental variables are
not exported again, and thus original method to export BB_TASKHASH doesn't apply
now which breaks all sstate packages. Now we can set data variable directly instead.
Signed-off-by: Kevin Tian <kevin.tian@intel.com>
If the checksum check failed, the .md5 stamp file would still have been created
meaning subsequent builds would proceed with the corrupt file. Reorder the calls
to avoid this. Also raise a specific error for the checksum not specified error
case.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This patch add the per-recipe SRC_URI checksum verification.
- SRC_URI format
The format of SRC_URI checksum follow OE definition:
1. SRC_URI has single src
SRC_URI = "http://some.domain/file.tar.gz"
SRC_URI[md5sum] = "xxxxxxxxxxxxxxx"
SRC_URI[sha256sum] = "xxxxxxxxxxxxxxxxxxxxxx"
2. SRC_URI has multiple src, every src need specify name
SRC_URI = "http://some.domain/file1.tar.gz;name=name1 \
http://some.domain/file2.tar.gz;name=name2 "
SRC_URI[name1.md5sum] = "xxxxxxxxxxxxxxx"
SRC_URI[name1.sha256sum] = "xxxxxxxxxxxxxxxxxxxxxx"
SRC_URI[name2.md5sum] = "xxxxxxxxxxxxxxx"
SRC_URI[name2.sha256sum] = "xxxxxxxxxxxxxxxxxxxxxx"
- SRC_URI checking invocation:
the checksum checking is invoked in do_fetch phase,
so it can be invoked manually by
# bitbake -f -c fetch <recipe_name>
if recipes has no SRC_URI checksum item, bitbake will show warning:
"
WARNING: Missing SRC_URI checksum for xxxx.tar.gz, consider to add
SRC_URI[md5sum] = "5c69f16d452b0bb3d44bc3c10556c072"
SRC_URI[sha256sum] = "f4e0ada8d4d516bbb8600a3ee7d9046c9c79e38cd781df9ffc46d8f16acd1768"
"
thus recipe author can add it to recpie file after SRC_URI
- control variable BB_STRICT_CHECKSUM
when SRC_URI checksum is missing, this variable decide pass or not
if BB_STRICT_CHECKSUM = "1", bitbake should fatal in this case, otherwise bitbake just pass
Signed-off-by: Yu Ke <ke.yu@intel.com>
sstate hash validation is done at initialization of RunQueueExecuteScenequeue.
However the index of 'valid' list returned from the validation doesn't
correspond to setscene task ID. It's just an intermediate namespace between
runqueue and sstate hash func. Use it as setscene task ID fully mess the flow.
Previously this doesn't cause trouble because all setscene tasks are passed. Commit
58396a5d24 add 'noexec' concept to setscene
tasks which grabs some tasks out of the list and thus trigger this problem
Without this fix there're ~50 recipes (gzip-native, glib, ...) rebuilt weirdly
with a minimal build, even though existing sstate packages could accelerate them.
there's another typo using wrong task ID in a debug message which further hide
this issue
Signed-off-by: Kevin Tian <kevin.tian@intel.com>
Add a parameter to getVarFlag() to auto-expand the value of the flag. This
makes getVarFlag() more consistent with getVar(), and allows expansion of
vardeps and vardepsexclude (which has been done in this commit).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Adds a vardepsexclude flag that can be used to exclude a dependency of
a variable (the opposite of vardeps). This will allow the exclusion of
variables from the hash generation much more selectively than blanket
whitelisting using BB_HASHBASE_WHITELIST.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Currently, anything whitelisted in the environment makes it into the worker
processes. This is undesireable and the worker environment should be as
clean as possible. This patch adapts bitbake sosme variables are loaded into
bitbake's datastore but not exported by default. Any variable can be exported
by setting its export flag.
Currently, this code only finalises the environment in he worker as doing so
in the server means variables are unavailable in the worker. If we switch
back to fork() calls instead of exec() this code will need revisting.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
idle command processing in each of the servers does not handle an explicit
None return value, which means the goggle UI ends up repeatedly adding
"Tasks Summary:" rows to the list.
This patch modifies BBCooker.buildTargets.buildTargetsIdle to return False
when BuildCompleted is fired, as is done in BBCooker.buildFile.buildFileIdle.
It may be that the correct way to fix this is to change the idle command
processing in the servers.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Construct a ProgressBar and pass it to the RunningBuild.handle_event() so
that goggle users are notified of metadata parsing progress.
UI's with status make users less nervous
Signed-off-by: Joshua Lock <josh@linux.intel.com>
It seems likely that the user would want to view the most recently emitted
messages so this patch sets the message dislay treeview to scroll to any
newly added rows.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Python 2.7's library changes some of xmlrpclib's internal implementation such
that interacting with a proxy to BitBakes SimpleXMLRPCServer would cause
BitBake to crash.
The issue was traced to changes in the xmlrpclib.Transport implementation and
Python bug #8194 (http://bugs.python.org/issue8194).
This patch introduces a workaround by create a subclass of
xmlrpclib.Transport, which overrides the offending methods with the Python
2.6.6 implementation copy and pasted from the Python 2.6.6 xmlrpclib, and
using this BBTransport implementation for both xmlrpclib.Server objects we
create.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
ProgressBar will be useful in other UI elements so make it it's own class.
Make ProgressBar a subclass of gtk.Dialog, rather than gtk.Window, so that we
can suggest the window manager parent the ProgressBar to the widget passed at
as parent.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Due to some recent change *somewhere* we need to explicitly look at the
name attribute on the instances class, rather than the name attribute of
the instance.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Move the test for ignored messages to the start of the message handling loop to
avoid doing work for messages which are only going to be ignored.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Some of the events we where trying to look for have had their class names
changed, fix these references.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
BB_TASKHASH_WHITELIST is expected to filter out native tasks from the
dependency list for target recipe's checksum. However current code
actually implements the opposite. All native sstate packages end up
to have empty task dependency while target sstate packages still have
native tasks counted into the checksum.
Signed-off-by: Kevin Tian <kevin.tian@intel.com>
Where a variable name consisted of an append/prepend combined with an override
and there was also an append/prepend to the variable, the override could be lost
if the override was not in OVERRIDES.
For example:
FOO = "A"
FOO_append = "B"
FOO_append_virtclass-native = "C"
could result in "AB" even though virtclass-native was in OVERRIDES.
With this patch applied, the result is "ABC" as would be expected.
The problem was the deletion of the _append/_prepend flag was happening
if *any* append/prepend was procesed, the result should really be that
it should contain any unprocessed append/prepend.
Kevin Tian deserves credit for looking into this and working out the
problem here.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Now 'runtaskdeps' is assigned before taskhash whitelist is filtered out, and thus
always contains original task dependencies. This causes problem to diagnose sstate
related problem. So fix it by appending filtered dep to 'runtaskdeps' in the fly.
Signed-off-by: Kevin Tian <kevin.tian@intel.com>
This means the noexec messages are only shown once as the stamp files are now
correctly created.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
two dep lists should be sorted before the comparison, or else it just
causes false-positive confusion.
Also fix 'taskdeps' to 'runtaskdeps' when printing out the difference.
Signed-off-by: Kevin Tian <kevin.tian@intel.com>
If sstate was used to accelerate a build, the pseudo directory might not have
been created leading to subsequent task failures.
Also, sstate packages were not being installed under pseudo context meaning
file permissions could have been lost.
Fix these problems by creating a FAKEROOTDIRS variable which bitbake ensures
exists before running tasks and running the appropriate setscene tasks under
fakeroot context.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
The git fetcher was failing to pull in new branches into a git
repository mirror tarball as the git fetch command being used didn't
add new remote branches.
This patch uses "git fetch --all" for fullclones to ensure any
new remote branches are cloned correctly.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
The more we try and patch up the fetcher code, the more things break. The
code blocks in question are practically unreadable and are full of corner
cases where fetching could fail. In summary the issues noticed included:
a) Always fetching strange broken urls from the premirror for "noclone"
git repositories
b) Not creating or rewriting .md5 stamp files inconsistently
c) Always fetching git source mirror tarballs from the premirror even
if they already exist but the checkout directory does now
d) Passing "None" values to os.access() and os.path.extsts() checks under
certain circumstances
e) Not using fetched git mirror tarballs if the preexist and always
try and fetch them.
This patch rewrites the sections of code in question to be simpler and
more readable, fixing the above problems and most likely other odd
corner cases.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Per function logging made it hard to track down what was happening so
switch to logging on a per task basis.
[BUGID #383]
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
We already export http,ftp,https proxy environment variables. Some environments,
GNOME for instance, place the socks proxy in ALL_PROXY and all_proxy. Export it
as well.
Signed-off-by: Darren Hart <dvhart@linux.intel.com>
Eric BENARD
Chris Larson
Khem Raj
Richard Purdie
C Michael Sundius
Chris Larson
Yu Ke
Joshua Lock
Kevin Tian
Paul Eggleton
Darren Hart