We're going to need a fakeroot/pseudo version of the worker so
abstract the code to start the worker process.
(Bitbake rev: b5d0f12f9df3ab211700473ed145ee6fbd9ca8e9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The worker was being executed by each execution queue so would get
constructed twice for each build. This is wasteful so move execution
to the main runqueue so we only have to start the worker once.
(Bitbake rev: 8117f8480125b121b2b5ac0afc31b108d9e670ae)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a pretty fundamental change to the way bitbake operates. It
splits out the task execution part of runqueue into a completely
separately exec'd process called bitbake-worker.
This means that the separate process has to build its own datastore and
that configuration needs to be passed from the cooker over to the
bitbake worker process.
Known issues:
* Hob is broken with this patch since it writes to the configuration
and that configuration isn't preserved in bitbake-worker.
* We create a worker for setscene, then a new worker for the main task
execution. This is wasteful but shouldn't be hard to fix.
* We probably send too much data over to bitbake-worker, need to
see if we can streamline it.
These are issues which will be followed up in subsequent patches.
This patch sets the groundwork for the removal of the double bitbake
execution for psuedo which will be in a follow on patch.
(Bitbake rev: b2e26f1db28d74f2dd9df8ab4ed3b472503b9a5c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The reasons for combining these objects is ancient history, it makes
sense to clean things up and separate them out now. This follows on
logically from the configuration cleansups and leads well into the
bitbake-worker changes.
(Bitbake rev: 89ffd62661ebcf2a97ce0c8dfd5e4d5bfbe27de7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This means the variables show up in the shell execution "run" files since
its useful to know what the fakeroot environment is and how to set it up
manually.
(Bitbake rev: bdf437747b664479acde6deaa9096e2a6bcdf483)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The Cooker class is too large and needs to be split up into different
functional units. Splitting out the collections code into its own class
seems like a good place to start to try and disentangle things.
(Bitbake rev: ca1fcbb6e214c155a05328779d3d326e10c5eac0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* dep variable was removed in
commit 3190cb83e2af195a464f669c5aa8aedbf795160e
Author: Richard Purdie <richard.purdie@linuxfoundation.org>
Date: Wed Jun 27 11:04:06 2012 +0100
taskdata: Add gettask_id_fromfnid helper function
This is like gettask_id but doesn't require translation of fnid -> fn
first which the function then translates back. This gives a sizeable
performance improvement since a significant number of lookups are avoided.
* now it fails completely instead of showing which task is missing
(Bitbake rev: 58847fabd389e5b8d02d5a9c6827aabedb30312f)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the setscene code was originally written it was thought that we'd
allow "partial" coverage. For example, if we just want to build the target
"bash:do_populate_sysroot" and its available from sstate, it makes no sense
to install gcc-cross's sstate package as its simply not needed.
Due to various other issues in the codebase, this functionality was
disabled/removed to allow the setscene code and sstate to stabilise and allow
us to concentrate on other problems.
The time has now come to enable "partial" coverage. There are two major changes
in this patch:
a) Creation of an unskippable list. This lists direct dependencies of
build targets and hence things that cannot be skipped.
b) Addition of a handler which looks at a given setscene target and what depends
on it and then decides whether its necessary.
(Bitbake rev: 2a937cd6a6c3110030b40bc4d85e349b85cb4db7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Mainly intended for the purpose of debugging or forcing builds
from source, the --no-setscene will prevent any setscene
tasks from running.
(Bitbake rev: 440e479f3e248482c38c149643403c6907ac7034)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a target dependency is marked as failed and yet we are continuing on
because -k has been specified, don't try to access the dependency's data
in taskData.build_targets since it will have been removed. This fixes
"IndexError: list index out of range" errors in this situation.
Also, do not print the "unhandled exception" message when SystemExit is
raised since we will have reported the actual error already in this
case (e.g. when -k has been specified and some targets failed).
Fixes [YOCTO #3133].
(Bitbake rev: 70eebc184eb1ab3678be87bed019b5beadecdc89)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The stamp files can change during setscene and the cache should be cleared to
account for this.
(Bitbake rev: 5ec12f586a50fce675b268965b3dc487aaa96c43)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was some odd behaviour if some task was run from setcene whilst there were
existing valid stamps for a depepdency. For example, do_populate_sysroot might
be installed at setscene time but if there were other tasks not installed from
setscene such as do_populate_lic which depend on do_configure, the setsceneverify
function would think that do_configure needed to be rerun and would hence void the
do_populate_sysroot and force that to rerun too.
The setsceneverify function needs to know which tasks are going to be rerun, not just
what the overall task list is and what setscene functions have run. This patch adds
that information and maintains backwards compatibility in a slightly ugly but effective
way. The metadata needs updating to take advantage of this change.
(Bitbake rev: 1423aafff97f17169e95ec3ba973eb002ff98c1c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The scenequeue code could result in one setscene function masking out another
which had been marked as notcovered. This change ensures the notcovered list
is taken into account when deciding which tasks need to be run, ensuring
that tasks that should be run do get run.
(Bitbake rev: 95b31127d8ba12ccb061d67481ece134c18c39f2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we call the hashvalidate for anything without a valid setscene stamp.
This improves the code to account for existing stamp files so that we only process
setscene hashes for things we might actually end up using. This avoids hash processing
and makes things slightly more efficient and the logs less confusing.
(Bitbake rev: 2e71a65bd910e8be894b8284eb0a50de7270c081)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was a bug where an exception in the runqueue code would cause an infinite loop
of debug messages. The exception would get reported but would loop since runqueue was
still registered as an idle handler.
This patch adds an exception handler to ensure in the case of errors, the system
more gracefully shuts down and doesn't loop.
(Bitbake rev: 99467c7e387f6e71358b40b8afae450d72cd68e0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We check the stamp cache before comparing t2 and t3 which means that we can miss
a level in the stamp file chains. The result of this is that a stamp can be accepted
as valid when in fact it isn't. Some weird behaviour alerted me to this in a local
build.
This patch also fixes to only uses the cache in recurse mode, there was a corner
case where stamps not in recurse mode could get added to the cache which could cause
an issue potentially.
(Bitbake rev: 7a5f776d325c0e37f58003424beda7cae2695325)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases we want to pull in DEPENDS and RDEPENDS of recrdeptask
dependencies but we need a way to trigger or avoid this behaviour
depending on context. The logical syntax to trigger such behaviour
would be a self referencing recrdeptask:
do_a[recrdeptask] = "do_a do_b"
The dependency chains already recurse this kind of expression correctly, the
missing piece is to avoid any circular reference errors. Since the dependencies
have already been recursively resolved, simply removing any recrdeptask
references is enough to break the circular references.
This patch therefore removes any circular references using the set
difference_update() operator. There will be metadata tweaks required to
add any references needed to the extra taskname.
(Bitbake rev: a5324da9b8a0c9307a6c511ea9009f34be70c92b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I'm not sure why we don't currently allow multiple entries in rdeptask when
we do in deptask. This makes the handling match between the two since
its trivial to fix.
(Bitbake rev: 19c84fe8854639768c874cc1449963a9867ad397)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, recrdepends is extremely greedy. For example:
do_foo[rdepends] = "somedep:sometask"
addtask foo
which adds foo with *no* dependencies, will suddenly start appearing
as a dependency in every task which uses recrdepends. So far this has
been mildy annoying but we now have use cases where this makes no sense
at all.
This reworks the recrdepends code to avoid this problem. To do this we
can no longer collapse things into lists just based on file ID. The problem
is this code is extremely performance sensitive. The "preparing runqueue"
phase spends a lot of time in these recursive dependency calculations so any
change here could negatively impact the user experience.
As such, this code has been carefully tested on convoluted dependency trees
with operations like "time bitbake world -g". The net result of this change
and the preceeding changes combined is a net speed up of these operations in
all cases measured.
Tests were made comparing "bitbake world -g" task-depends.dot before and after
this patch. There *are* differences for example -nativesdk do_build dependencies
on -native recipes are no longer present. All removed dependencies appear to
be sensible improvements to the system. The "rdepends" cross contamination
issue above is also fixed.
(Bitbake rev: 82d73423c57569b984ee0ae3d93e3c3bd5dc5216)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This gives some small performance gains and sets the scene for other
improvements by removing the need for duplicate detection code.
(Bitbake rev: 6fd723479e8d49227fd58040b3485c1d5afc4bc5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is like gettask_id but doesn't require translation of fnid -> fn
first which the function then translates back. This gives a sizeable
performance improvement since a significant number of lookups are avoided.
(Bitbake rev: 3190cb83e2af195a464f669c5aa8aedbf795160e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently its not possible to add arbitrary RDEPENDS to a specific task.
This can be useful and this patch adds functionality equivalent to the
'depends' task flag.
(Bitbake rev: db65080a6199baecc5c422294a4c4a9fd12dc29e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This new command line option forces the specified task and all dependent
tasks up to the default task to re-run. This means that the following
single step:
bitbake -C compile somerecipe
is equivalent to the following two steps (with the recent change to -f):
bitbake -c compile -f somerecipe
bitbake somerecipe
Note that to work this option needs full hashing enabled (i.e.
BB_SIGNATURE_HANDLER must be set to a signature handler that inherits
from BasicHash). If this is not the case, -C effectively does nothing.
Based on a previous implementation of this option by Jason Wessel
<jason.wessel@windriver.com>.
Implements [YOCTO #2615].
(Bitbake rev: 2530e0faada5775897cfd1b93aba6925826dca73)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If -f is specified, force dependent tasks to be re-run next time. This
works by changing the force behaviour so that instead of deleting the
task's stamp, we write a "taint" file into the stamps directory, which
will alter the taskhash randomly and thus trigger the task to re-run
next time we evaluate whether or not that should be done as well as
influencing the taskhashes of any dependent tasks so that they are
similarly re-triggered. As a bonus because we write this file as
<stamp file name>.taskname.taint, the existing code which deletes the
stamp files in OE's do_clean will already handle removing it.
This means you can now do the following:
bitbake somepackage
[ change the source code in the package's WORKDIR ]
bitbake -c compile -f somepackage
bitbake somepackage
and the result will be that all of the tasks that depend on do_compile
(do_install, do_package, etc.) will be re-run in the last step.
Note that to operate in the manner described above you need full hashing
enabled (i.e. BB_SIGNATURE_HANDLER must be set to a signature handler
that inherits from BasicHash). If this is not the case, -f will just
delete the stamp for the specified task as it did before.
This fix is required for [YOCTO #2615] and [YOCTO #2256].
(Bitbake rev: f7b55a94226f9acd985f87946e26d01bd86a35bb)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This function was used by old code such as packaged staging but is thankfully
obsolete now and replaced with better mechanisms. Its time to remove it and
the horrible internal only variables associated with it.
(Bitbake rev: 2995b8d551e0532eca20f8862730acd062c608ad)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was originally used to check the consistency of the stamps in one function
call. This turns out to be inefficient, unnecessary and if it were necessary,
check_stamp_task() could be called in a loop. The function has been unmodified
for a while and likely contains bugs. Its best simply removed.
(Bitbake rev: 728ffde1bd69b880d48fe8523b1616956d13616e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This should fix issues where bitbake would seemingly lock up when checking
certain configurations of stampfiles.
The cache is kept within the runqueue since that feels like the right
place to associate this cache data.
(Bitbake rev: e95755c4931b26d9f8102ed3652dff969145cfc9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- for a normal exit, use WEXITSTATUS, rather than manually shifting
- for exit via signal, set the exit code to 128+N, per shell convention
- if a process was stopped, return and don't handle it, as the process can yet
be continued
This should fix the case where bitbake says a task failed with an exit code of
0 (we assumed failure based on the overall status, but didn't pass all the
information along to task_fail).
(Bitbake rev: 84ea614bc56d35a414eb5bf5658891b340bfc569)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When "Force stop" is performed during the build stage, after os.kill() kills the build sub-processes, there are many <defunct> python processes in the system. In Hob, when the user initiates a new build, os.waitpid() in runqueue_process_waitpid() will be called, and the pids of those <defunct> processes will be returned as result[0], then self.build_pids[result[0]] will throw KeyError exception because now for the new build self.build_pids is empty.
This patch is to address the above issue to collect the results and handle the sub-processes as soon as they are killed.
[Yocto #2186]
(Bitbake rev: e9f4ca467e795bbc520d12b0e7a5985b6ff0a20e)
Signed-off-by: Shane Wang <shane.wang@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: c4c8df0e1e77f99f3ee0b07e1748116fc362819f)
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the code checking whether stamps are valid only traverses one step
of the dependency graph. This works fine in the normal cases where we've already
validated dependencies but for the setscene code, it doesn't work well. A typical
problem usecase is something like:
bitbake gcc-cross -c unpack -f
bitbake gcc-cross
which will ignore any sstate files already cached which could be used to speed
up the gcc-cross build. This becomes particularly problematic with multiple gcc
builds where only one should rebuild yet they all do.
This patch teaches the stamp code to be able to recurse within a given fn
which gives the behaviour people would expect from the code and allows
bitbake to make better use of prebuild sstate objects.
(Bitbake rev: e68814cb2e8da523d4ddf05e8ceddfaa19244851)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes a regression introduced in commit
e8a3499c95a6d4f2b8fed002fb9504733c5be3c6 which resulted in a
backtrace on setscene task failure due to trying to dereference
the setscene task ID twice.
(Bitbake rev: 8b846a92a58b5c20d7cfd2efd32b763e95c3c2fd)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sometimes user stops the build before runqueue is established,
for example, at the stage of running add_unresolved() function. This
will cause RunQueue to use rqexe field in finish_runqeue() before
initialized. This will cause endless print of "Running idle function"
if use process server.
This commit initialize rqexe variable in RunQueue's init function,
and add a judgement in finish_runqueue().
(Bitbake rev: 59f817723172092a87738c79f555e605f55ea375)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* When a setscene task starts, print out that it's starting in the UI
(ensuring we get the correct task name)
* When a setscene task fails, ensure we remove it from the list of
running tasks so that if you break out any time afterwards it
is not still listed.
(Bitbake rev: e8a3499c95a6d4f2b8fed002fb9504733c5be3c6)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make dry-run do everything except executing the task, instead of
cutting it off earlier. This fully tests the code path for running the
child task (parsing and fakeroot), as well as enabling future
functionality such as using dry-run to produce signature files.
(Bitbake rev: bf1d7739618dabf3872a868230c0112b9ad2a2c0)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Monitor disk availability and take action when the free disk space or
amount of free inode is running low, it is enabled when BB_DISKMON_DIRS
is set.
* Variable meanings(from meta-yocto/conf/local.conf.sample):
# Set the directories to monitor for disk usage, if more than one
# directories are mounted in the same device, then only one directory
# would be monitored since the monitor is based on the device.
# The format is:
# "action,directory,minimum_space,minimum_free_inode"
#
# The "action" must be set and should be one of:
# ABORT: Immediately abort
# STOPTASKS: The new tasks can't be executed any more, will stop the build
# when the running tasks have been done.
# WARN: show warnings (see BB_DISKMON_WARNINTERVAL for more information)
#
# The "directory" must be set, any directory is OK.
#
# Either "minimum_space" or "minimum_free_inode" (or both of them)
# should be set, otherwise the monitor would not be enabled,
# the unit can be G, M, K or none, but do NOT use GB, MB or KB
# (B is not needed).
#BB_DISKMON_DIRS = "STOPTASKS,${TMPDIR},1G,100K WARN,${SSTATE_DIR},1G,100K"
#
# Set disk space and inode interval (only works when the action is "WARN",
# the unit can be G, M, or K, but do NOT use the GB, MB or KB
# (B is not needed), the format is:
# "disk_space_interval, disk_inode_interval", the default value is
# "50M,5K" which means that it would warn when the free space is
# lower than the minimum space(or inode), and would repeat the action
# when the disk space reduces 50M (or the amount of inode reduces 5k)
# again.
#BB_DISKMON_WARNINTERVAL = "50M,5K"
[YOCTO #1589]
(Bitbake rev: 4d173d441d2beb8e6492b6b1842682f8cf32e6cc)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we call finish_now(True), rq.state is not updated to match. This
makes the behaviour of finish_now(False) and finish_now(True) consistent
so both leave rq.state consistently.
(Bitbake rev: 9079ae0ab74f9232b7e9853b2013b051d4fcf623)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit implements a new design for hob
Some of the new features:
- Friendly new designed GUI. Quick response to user actions.
- Two step builds support package generation and image generation.
- Support running GUI seprarately from bitbake server.
- Recipe/package selection and deselection.
- Accurate customization for image contents and size.
- Progress bars showing the parsing and build status.
- Load/save user configurations from/into templates.
(Bitbake rev: 4dacd29f9c957d20f4583330b51e5420f9c3338d)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Shane Wang <shane.wang@intel.com>
Signed-off-by: Liming An <limingx.l.an@intel.com>
Signed-off-by: Fengxia Hua <fengxia.hua@intel.com>
Designed-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current code prints a log when a setscene task starts, therefore
the progressbar in hob will not receive it. Use a sceneQueueTaskStarted
event instead.
Besides, change the sceneQueueTaskFailed event to inherit runQueueEvent
directly to avoid confusion to event receiver.
(Bitbake rev: 7c07cc93d6558d7d9c3144b13493901b7ebae050)
Signed-off-by: Shane Wang <shane.wang@intel.com>
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When building the universe target:
* Show a warning about universe likely producing errors
* Multiprovider errors are now shown as warnings
Also remove an unused "error" variable in runqueue.py.
Fixes [YOCTO #1936]
(Bitbake rev: 293c796e8a3d1f74ca1e51017b6dede261612281)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is being added to resolve setscene race issues where we do have
particular dependencies required between setscene tasks. This allows
specific dependencies to be specified. This allows us to fix the races
in sstate with the useradd class in OE-Core.
Any tasks being depended upon have their reverse dependencies cleared to
ensure we don't have circular references.
(Bitbake rev: e1b157d26374a70e6274edcb4c0b9f3bc48f765c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When constructing the setscene inter-dependencies, we need to account for all task,
not just the last one found. This patch corrects this oversight and ensures all
dependencies are added, not just the first one found.
(Bitbake rev: b9b5b5129d066e1ff7d3effda116afc3c6657beb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It's unclear from the history but at some point we stopped logging the
"Tasks Summary" NOTE when tasks failed. Reinstate this for failure, and
also make the count of attempted tasks include the failed task.
(Bitbake rev: e44d5be98fc5d2589cd929ce143638395936b936)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Seeing the word "failed" alone without reading the whole context has
occasionally triggered an automatic assumption on the part of some users
(myself included) that something has gone wrong, even when this message
is telling you that "0 [tasks] failed". To avoid this let's just say
"all succeeded" in this case instead.
As a bonus this means you can now search the output for "fail" and not
find anything if all went well.
(Bitbake rev: b6f067af12d4661758a78788f1db472684b9aba8)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When debugging setscene problems it was found that some extra debug messages were useful.
This patch adds them.
(Bitbake rev: cff2c258b77fde01d530a5923e553e6111b15eb5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The existing looping code can mask an existing "found = True"
by forcing it to False each time. This can lead to dependency
lists not being fully searched and results in dependency errors.
An exmaple of this was the autobuilder building linux-yocto from
sstate but then rebuilding some of the recipe's tasks for no
apparent reason. Separating the logic into two variables solves this
problem since any "found = True" value is now always preserved.
(Bitbake rev: 61017fc5d30b7a13308d038872ec92efc1a84cef)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If A depends upon B which depends upon C and the setscene for B
succeeds but C is going to get rebuilt, we should wait for C to
try and build A but currently we don't.
This is due to the timing of when we run the task_skip() as this
triggers other tasks to become buildable. This patch moves the timing
of that call to a more appropriate place allowing dependencies to
behave as expected.
(Bitbake rev: b7114d8e5d9b0720339bd5d24d243c0f2a7c1f3b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The OE metadata has cases where it needs to prevent some setscene
functions from running. An example of this is where we know a task
is going to run do_configure (which would clean out do_populate_sysroot)
and hence we don't want do_populate_sysroot_setscene to run.
This change adds in a hook so that the metadata can allow any
such policy decision to filter back up to bitbake.
It removes the existing code which attempted to do this in a generic
way but failed.
(Bitbake rev: 2f6d987d9957a5d713ce119c24c2e87540611f53)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code which populates setscene_covered list was adding a task to the
covered list if all of the tasks that depend upon it were also covered;
however, this means that tasks that would have installed "runtime"
dependencies were being marked as covered also, e.g. gmp-native and
mpfr-native are needed by gcc-cross at runtime since they are shared
libraries that gcc links to, but their do_populate_sysroot tasks were
being marked as covered, resulting in failures later on if gcc-cross was
available from sstate but mpfr-native and gmp-native weren't.
Since we currently have no real way to handle runtime dependencies for
native packages, add a workaround which avoids marking tasks as covered
if one or more of their revdeps are from a different recipe.
Fixes [YOCTO #1536].
(Bitbake rev: e492eb4dc9016cd0bed194377c6f2b85cf0ad113)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a FAKEROOTNOENV which does the opposite of the FAKEROOTENV variable
and is data loaded into the environment for tasks without the fakeroot
flag.
The intent here is to provide a way to control the environment when we
aren't needing fakeroot context which allows us to unload the preload
from memory entirely and gain a valuable speedup.
I'm not 100% happy with needing this at the bitbake level, particularly
with the cache hit but it does give a valuable speedup.
(Bitbake rev: 05c29ab0b2ad3c521414cabb6a92bca15c6e919c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this, variables can be set to one thing in one part of the environment and something
different in another part. This change ensures the datastore and the environment
are consistent.
(Bitbake rev: 459addf13721a6847406f215650fa1882fb83ea9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fixes two problems:
a) Variables which were in the parent environment but not set as "export"
variables in the datastore could end up in the task environment
b) oe.environ.update() can't cope with the generator returned by
bb.data.exported_vars()
Whilst the updated code isn't as neat, it does do the expected thing,
sets the environment correctly and stops unwanted values leaking into
the task environment.
(Bitbake rev: 67e5e23034c5ec2b9efcca935242830306c0048d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch, based on proof of concept code from Richard adds code to
bitbake to allow individual tasks to optionally specify their stamp
file using the stamp-base flag. This takes the same form as the STAMP
variable but can be specified on a per task basis.
Code is also added to runqueue to ensure that if two tasks share the
same stamp file, only one will be executed at once.
A significant usecase for this code is to share source code (${S})
between recipes where separate build directories (${B}) are used.
(Bitbake rev: 41bef02bef8379590ba012319aebe05068a8081e)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The umask for a task can now be set as:
task[umask] = 022
task[umask] = '022'
If specified as a text string, it must be octal. (This is due to
recipe parsing where it's always set to a string.)
[RP tweaked to use None instead of -1]
(Bitbake rev: a5caaaaba8f0db1af5d8f2e610021d6d4b56894e)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update exception handling syntax to use the modern style:
except ExcType as localvar
(Bitbake rev: dbf5f42b06bef81749b13aa99945cc1292a6676d)
Signed-off-by: Scott Garman <scott.a.garman@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
JaMa reported issues where bitbake would rebuild things instead of using the
existing built tasks. This was tracked to a case where:
a) rm_work is uses
b) A depends on B
c) B has a version change (e.g. PR bump)
and A *and* B would then rebuild.
It turns out that rm_work was correctly turning stamp files into the correct
_setscene varients but bitbake was then ignoring them during setscene processing.
If the correct sstate checksumed files didn't exist, everything would seemingly
rebuild.
The fix is to check for existing *_setscene stamps and if present, honour them.
If "basichash" is enabled, the hash is included with the stamps so everything
should then function as intended.
(Bitbake rev: 0a4d857aabc86b973170ba9ce32b6b449a4e2165)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a build causes a real task to be run when the setscene task has already
run then it was possible for dependent packages to be rebuilding at the same
time as a rebuild of the packages they depended on, resulting in failures
when files were missing. This change looks in the setscene covered list and
removes anything where a dependency of the real task is going to be run (e.g.
do_install is going to be run even though the setscene equivalent of
do_populate_sysroot has already been run).
As an additional safeguard we also delete the stamp file for the setscene
task under these circumstances.
Fixes [YOCTO #792]
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This patch is a quick proof of concept to show how source code could
be shared between recipes which use ${B} to have a separate build
directory compared to source directory ${S}.
Issues:
a) gcc uses sed and creates config files against ${S} which means
the directory should not be shared. Need to change the way that works.
b) Could be extended to cover eglibc except there is a patch applied
against nativesdk versions which again makes the source incompatible.
c) Need to clean up the layout in work-shared and make a directory level deeper
to ensure patch separation.
d) clean task does not remove stamps
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a setscene task failed previously it was showing an incorrect task
name in the error line. This patch ensures we show the correct name, also
including the "_setscene" suffix.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Parsing the recipe in the parent before forking off the child worker
can mean the parent doesn't hit the idle loop and becomes a bottleneck
when lauching many short lived processes.
The reason we need this in the parent is to figure out the fakeroot
environmental options. To address this, add the fakeroot variables
to the cache and move recipe loadData into the child task.
For a poky-image-sato build this results in about a 2 minute speedup
(1.8%).
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This avoids cases where the stats are modified after the event is fired but
before it's dispatched to the UI.
(Bitbake rev: 1954f182687a0bd429175dda87f05d8a94bb403a)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The server UI was reading 1024 bytes, then sleeping for 0.25 seconds. Since
most new LogRecord events are larger than this it leads to a build up of data
which is only processed slowly, leading to a bottleneck and a slow down of
all bitbake processes.
Thanks to Dongxiao Xu <dongxiao.xu@intel.com> for the great work in debugging
this. A large value has been left in for the read() command just to ensure some
fairness amongst process handling if a task tries to log truly huge amounts of
data to the server, or goes crazy and ensures the main loop doesn't stall.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For certain tasks, we need additional information in build stamp file
other than the task name and file name. stamp-extra-info is introduced as
a task flag which is appended to the stamp file name.
[Code simplifcations/tweaks from Richard]
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie
Martin Jansa
Jason Wessel
Paul Eggleton
Christopher Larson
Shane Wang
Robert P. J. Day
Dongxiao Xu
Robert Yang
Mark Hatle
Scott Garman