If a build causes a real task to be run when the setscene task has already
run then it was possible for dependent packages to be rebuilding at the same
time as a rebuild of the packages they depended on, resulting in failures
when files were missing. This change looks in the setscene covered list and
removes anything where a dependency of the real task is going to be run (e.g.
do_install is going to be run even though the setscene equivalent of
do_populate_sysroot has already been run).
As an additional safeguard we also delete the stamp file for the setscene
task under these circumstances.
Fixes [YOCTO #792]
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This patch is a quick proof of concept to show how source code could
be shared between recipes which use ${B} to have a separate build
directory compared to source directory ${S}.
Issues:
a) gcc uses sed and creates config files against ${S} which means
the directory should not be shared. Need to change the way that works.
b) Could be extended to cover eglibc except there is a patch applied
against nativesdk versions which again makes the source incompatible.
c) Need to clean up the layout in work-shared and make a directory level deeper
to ensure patch separation.
d) clean task does not remove stamps
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a setscene task failed previously it was showing an incorrect task
name in the error line. This patch ensures we show the correct name, also
including the "_setscene" suffix.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Parsing the recipe in the parent before forking off the child worker
can mean the parent doesn't hit the idle loop and becomes a bottleneck
when lauching many short lived processes.
The reason we need this in the parent is to figure out the fakeroot
environmental options. To address this, add the fakeroot variables
to the cache and move recipe loadData into the child task.
For a poky-image-sato build this results in about a 2 minute speedup
(1.8%).
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This avoids cases where the stats are modified after the event is fired but
before it's dispatched to the UI.
(Bitbake rev: 1954f182687a0bd429175dda87f05d8a94bb403a)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The server UI was reading 1024 bytes, then sleeping for 0.25 seconds. Since
most new LogRecord events are larger than this it leads to a build up of data
which is only processed slowly, leading to a bottleneck and a slow down of
all bitbake processes.
Thanks to Dongxiao Xu <dongxiao.xu@intel.com> for the great work in debugging
this. A large value has been left in for the read() command just to ensure some
fairness amongst process handling if a task tries to log truly huge amounts of
data to the server, or goes crazy and ensures the main loop doesn't stall.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For certain tasks, we need additional information in build stamp file
other than the task name and file name. stamp-extra-info is introduced as
a task flag which is appended to the stamp file name.
[Code simplifcations/tweaks from Richard]
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current parameters are not useful to the stampfile generator function
as they can't uniquely define a task. This updated things so the
parameters can identify unique tasks.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously, the cache was actually being loaded from disk twice whenever using
-b or -e -b. This also moves the bb_cache instance into the CookerParser, as
it's not needed by the cooker itself at all.
(Bitbake rev: dd0ec2f7b18e2a9ab06c499b775670516bd06ac8)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
range() allocates an actual list when called. xrange() is just an iterator
and creates the next range item on demand. This provides a slight
performance increase.
In python 3, range will do what xrange does currently, but the upgrade will
be handled by the 2to3 tool.
(Bitbake rev: 73b40f06444cb877a5960b2aa66abf7dacbd88f0)
Signed-off-by: Bob Foerster <robert@erafx.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Per the python documentation, os.waitpid returns the exitcode shifted up by 8
bits, and we weren't compensating, resulting in a display of 'failed with 256'
when a worker process exits with a code of 1.
(Bitbake rev: 90c2b6cb24dc9c82f0a9aa9d23f2d1ed2e6ff301)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
We use a custom Logger subclass for our loggers
This logger provides:
- 'debug' method which accepts a debug level
- 'plain' method which bypasses log formatting
- 'verbose' method which is more detail than info, but less than debug
(Bitbake rev: 3b2c1fe5ca56daebb24073a9dd45723d3efd2a8d)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This kills firing of Msg* events in favor of just passing along LogRecord
objects. These objects hold more than just level and message, but can also
have exception information, so the UI can decide what to do with that.
As an aside, when using the 'none' server, this results in the log messages in
the server being displayed directly via the logging module and the UI's
handler, rather than going through the server's event queue. As a result of
doing it this way, we have to override the event handlers of the base logger
when spawning a worker process, to ensure they log via events rather than
directly.
(Bitbake rev: c23c015cf8af1868faf293b19b80a5faf7e736a5)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
It needs to be a generator, so scheduler subclasses have the option to skip
buildable tasks and return a later one.
(Bitbake rev: a8c61e41bc6277222e4cde667ad0b24bd1597aa0)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
If you create a runqueue scheduler class in a python module, available in the
usual python search path, you can now make it available to bitbake via the
BB_SCHEDULERS variable, and the user can then select it as they select any
other scheduler.
Example usage:
In a test.py I placed appropriately:
import bb.runqueue
class TestScheduler(bb.runqueue.RunQueueScheduler):
name = "myscheduler"
In local.conf, to make it available and select it:
BB_SCHEDULERS = "test.TestScheduler"
BB_SCHEDULER = "myscheduler"
(Bitbake rev: 4dd38d5cfb80f9bb72bc41a629c3320b38f7314d)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
SIGINT should be from the user, not a script. It also doesn't work as
reliably to shut down processes, as it's not always interpreted as a
termination request. In addition, it causes KeyboardInterrupt exceptions in
the worker processes, which can interfere with our exception handling.
(Bitbake rev: e5f6e0e9de4c6d1dfdd269d2bf7f83c00c415a27)
Signed-off-by: Chris Larson <clarson@kergoth.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
In previous exec() model, cooker is re-initialized from scratch with environmental
variable exported accordingly. Now in fork() model, environmental variables are
not exported again, and thus original method to export BB_TASKHASH doesn't apply
now which breaks all sstate packages. Now we can set data variable directly instead.
Signed-off-by: Kevin Tian <kevin.tian@intel.com>
sstate hash validation is done at initialization of RunQueueExecuteScenequeue.
However the index of 'valid' list returned from the validation doesn't
correspond to setscene task ID. It's just an intermediate namespace between
runqueue and sstate hash func. Use it as setscene task ID fully mess the flow.
Previously this doesn't cause trouble because all setscene tasks are passed. Commit
58396a5d24 add 'noexec' concept to setscene
tasks which grabs some tasks out of the list and thus trigger this problem
Without this fix there're ~50 recipes (gzip-native, glib, ...) rebuilt weirdly
with a minimal build, even though existing sstate packages could accelerate them.
there's another typo using wrong task ID in a debug message which further hide
this issue
Signed-off-by: Kevin Tian <kevin.tian@intel.com>
Chris Larson
Paul Eggleton
Richard Purdie
Dongxiao Xu
Richard Purdie
Bernhard Reutner-Fischer
Bob Foerster
Chris Larson
Kevin Tian