If -f is specified, force dependent tasks to be re-run next time. This
works by changing the force behaviour so that instead of deleting the
task's stamp, we write a "taint" file into the stamps directory, which
will alter the taskhash randomly and thus trigger the task to re-run
next time we evaluate whether or not that should be done as well as
influencing the taskhashes of any dependent tasks so that they are
similarly re-triggered. As a bonus because we write this file as
<stamp file name>.taskname.taint, the existing code which deletes the
stamp files in OE's do_clean will already handle removing it.
This means you can now do the following:
bitbake somepackage
[ change the source code in the package's WORKDIR ]
bitbake -c compile -f somepackage
bitbake somepackage
and the result will be that all of the tasks that depend on do_compile
(do_install, do_package, etc.) will be re-run in the last step.
Note that to operate in the manner described above you need full hashing
enabled (i.e. BB_SIGNATURE_HANDLER must be set to a signature handler
that inherits from BasicHash). If this is not the case, -f will just
delete the stamp for the specified task as it did before.
This fix is required for [YOCTO #2615] and [YOCTO #2256].
(Bitbake rev: f7b55a94226f9acd985f87946e26d01bd86a35bb)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When an ExpansionError occurs during parsing it is useful to know which
recipe was being parsed when it occurred.
(Bitbake rev: a2a8e8c9f5a1a30aacab696fd2b434939ee2c4e2)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix a recent regression where we see the following additional error
after an error occurs during parsing:
ERROR: Command execution failed: Traceback (most recent call last):
File "/home/paul/poky/poky/bitbake/lib/bb/command.py", line 84, in runAsyncCommand
self.cooker.updateCache()
File "/home/paul/poky/poky/bitbake/lib/bb/cooker.py", line 1202, in updateCache
if not self.parser.parse_next():
File "/home/paul/poky/poky/bitbake/lib/bb/cooker.py", line 1672, in parse_next
self.virtuals += len(result)
UnboundLocalError: local variable 'result' referenced before assignment
(Bitbake rev: 1ae0181ba49ccfcb2d889de5dd1d8912b9e49157)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
[Yocto 2142]
Force to exit HOB when hob is parsing recipes, the bitbake doesn't stop.
It hangs on function BitBakeServerConnection::terminate in file
server/process.py:
else:
self.procserver.join()
It is waiting for the children process quit.
In stage of parse recipes BBCooker spawns Parser processes as many as
cpu numbers. When quit the Parser processes they make their internal
Queue to call cancel_join_thread() to avoid block but don't work at
this time.
So force to terminate the Parser processes.
(Bitbake rev: bebef58b21bdff7a3ee1fa2449b7df19144f26fd)
Signed-off-by: Kang Kai <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We should always be passing full pathnames around within bitbake. If
a file was referenced as a relative path to the current working
directory, it might not get passed through the abspath call and
hence the cwd would not get added as a prefix.
This change adds a second pass at ensuring we only return absolute paths.
(Bitbake rev: 72fc62ca124a24e2dbe404a3c83a49608a7c7931)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Gathers a list of paths to have checksums calculated at parse time, and
processes these when calculating task hashes. Checksums are cached with
the file's current mtime. Thus, changing any local file in SRC_URI will
now cause the do_fetch taskhash to change, thus forcing a rebuild.
This change adds very roughly about an 8% increase in parse time (a few
seconds) and maybe a few seconds during runqueue generation, so a fairly
moderate performance hit.
Note that since paths are resolved at parse time, this will not force
a rebuild when files are introduced which would cause that resolved path
to be different - for example, where a machine-specific version of a file
was added without otherwise changing the recipe. This will need to be
handled in a future update.
Code to hook this into the signature generator was courtesy of
Richard Purdie <richard.purdie@linuxfoundation.org>.
Implements [YOCTO #2044].
(Bitbake rev: c993b7c457f8b7776e8a5dff253bfa0724bc2cae)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Inherit information could be used to judge if a recipe is image type
or not.
(From Poky rev: 203f112a84b3971dc71da3dc455fd054d1c16a12)
(Bitbake rev: fe52af99918f61abec8f779efb02926b713f2aac)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This means that statements that expand to more then one entry
such as:
CLASSES = "a b"
inherit ${CLASSES}
work correctly instead of trying to inherit a class called "a b".
(Bitbake rev: 2568e9ace6e6f483e1bf2a9ef2f4d8318d6c85b7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When calculating the image installation dependency, we need the
rdepends and rrecommends information of certain packages.
(Bitbake rev: 5eab31ad9e9431e27725dcefa38cbd7f2e37d394)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The parsing shutdown call can occur more than once. Currently
if this happens the code will hang. Add some code to prevent this.
[YOCTO #2039 partial]
(Bitbake rev: 7614c5bdcaf9d70614b4fad2ca02c7e6eaa92f1b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Remove part of the original logic that stores pkg dependency and pkg
recomends, which will not be used in Hob.
Add the judgement for preferred fn provider of a certain package,
which maps package name to its providing recipe name.
The above approaches correct the build dependency calculation,
and they also reduced the total depend_tree size, which speed up the
tree data generation time.
(Bitbake rev: d668eb1300b3b3115964e98127b1bef554caae17)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fnid is an outer variable and shouldn't be overrided, change the inner variable
name from 'fnid' to 'id'.
(Bitbake rev: c6dc8d6310af22c95adb06c64339e7ec9eaeb315)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Remove the "resolve" parameter since the original resolve=False
option is no longer be used.
(Bitbake rev: dadce609149cfb09ecdc53bfe1f416a3f57a5033)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Enable configuring whether "set +x" is added to all shell tasks rather
than forcing it; this is enabled by setting BB_VERBOSE_LOGS to 1.
(Bitbake rev: 659411b6bb30e1a8355afc1c29b8170a8f2b55ac)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Added new variable BBINCLUDED indicating the file dependency
information. It exposes the internal variable '__base_depends' and
'__depends'.
(Bitbake rev: af524a656fce32d01687481b86c31bef00eb9fc3)
Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we find an error in the layer configuration (such as an unsatisfied
item in LAYERDEPENDS) then exit by raising an exception at the end of
handleCollections() (without producing a backtrace).
(Bitbake rev: c7486a09310fe63b1aa1b7b0bb9450f306b6093b)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The changes include:
- Clean some events in event.py
- Fire essential events for Hob2 to handle with more information.
- knotty changes
(Bitbake rev: 9ede881620c501574f014e600cea6947ea908ac2)
Signed-off-by: Shane Wang <shane.wang@intel.com>
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we have generateTargetsTree API, which is used to get
dependency information. However in that tree, there will be
"virtual/xxx" in depends fields. Therefore we add the resolve option
to replace it with its real providers.
Besides, for packages that provided by multiple recipes, we will find
their preverred provider.
(Bitbake rev: 28501612efdfc6ee47576cc90deb6e897883e7f5)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously we use the file time stamp to judge if a cache is valid.
Here this commit introduce a new method, which calculates the total
hash value for a certain configuration's key/value paris, and tag
it into cache filename, for example, bb_cache.dat.xxxyyyzzz.
This mechanism also ensures the cache's correctness if user
dynamically setting variables from some frontend GUI, like HOB.
(Bitbake rev: 1c1df03a6c4717bfd5faab144c4f8bbfcbae0b57)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ensure that a file and line number are reported for ParseError where
possible. This helps particularly in the case of inherit and require
which previously did not report either of these upon failure.
(Bitbake rev: f588ba69622a2df35417ced184e56c79ac1b40d5)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes a regression introduced in BitBake rev
c9f58ef6b897d3fa5b0d23734b5f2cb3dabb057a which prevents errors during
parsing from being fully reported because BitBake shuts down before it
can print them. Move the error printing to before the shutdown in order
to fix it.
Also remove a couple of superfluous calls to sys.exit() since these
will never be called (as self.shutdown(clean=False) does this itself).
(Bitbake rev: 356c42ed00541dea2c382c57f768a4f431d22db0)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When building the universe target:
* Show a warning about universe likely producing errors
* Multiprovider errors are now shown as warnings
Also remove an unused "error" variable in runqueue.py.
Fixes [YOCTO #1936]
(Bitbake rev: 293c796e8a3d1f74ca1e51017b6dede261612281)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The number of threads is self.num_processes - 1 due to the range(1,
self.num_processes). Fixed by changing to range(0, self.num_processes).
(Bitbake rev: adc041fd9e3def29cdf9c1ae4849c5383bac46e5)
Signed-off-by: Caleb Crome <caleb@signalessence.com>
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fixes the hang issue encountered with parse errors. The underlying issue
seems to have been the pool.terminate(). This sends SIGTERM to each of the
multiprocessing pool's processes, however, a python process terminating in
this fashion can corrupt any queues it's interacting with, causing a number of
problems for us (e.g. the queue that sends events to the UI).
So instead of using multiprocessing's pool, we roll our own, with the ability
to cancel the work. In the very long term, the python concurrent.futures
module introduced in python 3.2 could be used to resolve this as well.
(Bitbake rev: 7c39cfd8e060cca8753ac4114775447b18e13067)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without it, we get random hangs on parse failure. With it, some folks have
seen hangs even on successful cases. The former is clearly less problematic.
This is temporary, until I can finish investigating the root causes of both
issues.
(Bitbake rev: db689a99beffea1a285cdfc74a58fe73f1666987)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the addition of new cache domains, the summary, license and section
information is no longer requred in the core cache since its only used by
the hob UI. This patch removes the duplicated entries.
It also adds the DESCRIPTION field to the cache for the benefit of hob2.
(Bitbake rev: 33ffb2e99825cb643b148b3462c2d4cf33ff5f58)
Signed-off-by: Shane Wang <shane.wang@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make the following improvements to the show-overlayed subcommand:
* Show recipes that are overlayed when the version is higher or lower,
not just when it is the same. This gives a much better picture of the
influence each layer is having over the metadata used for building.
This can be disabled with the -s option if you just want to see
recipes with the same version as before.
* Default to showing name (PN), layer and version rather than the full
path and filename. The old style formatting can be used by specifying
the -f option.
* Mark skipped recipes as such in the output, and print them in the
correct sorted place in the list rather than at the end
* Prefix/suffix title line with === so it can be filtered out easily in
shell scripts if desired
(Bitbake rev: 43b473275d3cb2e60a14e4a52cdc4654b3f4e5e7)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By calling init_parser which sets up the siggen code after the ConfigParsed
event is fired, we can allow the metadata to add siggen classes which
was always what the code intended.
(Bitbake rev: 69fd5ad4a8402fab5eb8348260966b337c5648ff)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This avoids some silent parser hangs we were seeing which were
near impossible to debug as no user feedback was given.
[RP: Tweak commit message]
(Bitbake rev: d104f29871c04a5a36600a35b2568b49e5b21ca0)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Remove the error logged within cooker summarising the list of failed
tasks, and instead print this in the UI (knotty) where it belongs. This
also adds the actual name of the task that failed as well as the
corresponding recipe file that was being shown previously.
In addition, reformat the summary messages more tidily - no extra breaks
between lines and use correct English singular/plurals, with some
allowance for future translation.
(Bitbake rev: cdf69913f99d28bc7f51067a60257701f952c6cb)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Improves error output for matching problems when the -b / --buildfile
command line option is used.
Rename MultipleMatches exception to NoSpecificMatch (as it is also
raised when there are no matching recipes) and make it inherit from
BBHandledException so that it doesn't print a stack trace (we always log
an ERROR prior to raising it.)
In addition, improve the formatting of the error message - only call the
log function once rather than once for every match, and use a more
appropriate message if there are no matches.
Fixes [YOCTO #1141]
(Bitbake rev: 803550a5098ec878164245e71344c3d687310b72)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
[YOCTO #1126]
A local PR service will be started and stopped automatically along
with the bitbake invocation/ternimation.
This local PR service will be started only and if only when the
PRSERV_HOST is set to 'localhost' and PRSERV_PORT is set to '0'.
When started, the sqlite3 database is stored at
"${PERSISTEN_DIR}/prserv.sqlite3" or "${CACHE}/prserv.sqlite3".
(Bitbake rev: 9d8f45407c67ed0d3c4f820cf646de3c385067c7)
Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is no direct use of command in cooker.py, and it is using
bb.command instead. Remove command in the import list.
This fixes a problem of embedded import between command.py and
cooker.py.
(Bitbake rev: c353316b2efcc7a893d6b4aa9a9647d51a6f69e3)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is no direct use of command in cooker.py, and it is using
bb.command instead. Remove command in the import list.
This fixes a problem of embedded import between command.py and
cooker.py.
(Bitbake rev: 92fec2788e33d301cc63848901bc6adc764a2ecf)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
initCooker is to set the cooker to the initial state with nothing
parsed.
(Bitbake rev: 8ee9c6cc47938604268242d90bd3007218f9a36e)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For buildTargets function, we use bb.configuration.data as parameter
to inject events, since in hob environment, some variables are
modified dynamically and bb.configuration.event_data may out of date.
(Bitbake rev: 487a124942fd5cd9de71413b79a4049a6d26f1e8)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The assignments of self.configuration.cmd in BBCooker seems duplicated,
have the followings in both BBCooker::__init__ and
BBCooker::loadConfigurationData:
if not self.configuration.cmd:
self.configuration.cmd = self.configuration.data.getVar("BB_DEFAULT_TASK", True) or "build"
The __init__ invokes the loadConfigurationData, and it would make sure
that self.configuration.cmd has been assigned a proper value, so we can
remove the one in __init__.
[YOCTO #1791]
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using bitbake -e -b virtual:xxxx:/path/to/the.bb would result in
zero matches since the virtual:xxxx piece wasn't being processed.
This adds in the necessary functionality to handle it correctly.
[YOCTO #1793]
(Bitbake rev: bd5a727c8447bcb747c1d2463b7de2ab6d21a7de)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bitbake codebase makes assumptions that only one copy is active
against a given build directory at a given time. This patch adds a
lockfile in TOPDIR to ensure that is the case.
Note that no unlock is needed, that is automatically dropped when
execution terminates.
(Bitbake rev: 6004cbf36c980c2574c1c9153df81a7c55317e17)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake -s was not displaying correct version information when the
PREFERRED_VERSION string contains other variables. The actual built
versions would differ since the providers.py functions were called
with expanded keys at this point.
This patch expands keys for showVersions bringing everything into
sync correctly.
[YOCTO #1493]
(Bitbake rev: 3a0f2dda3c6de993f08ed50e9d513add9407339c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The recent change which modified inheritFromOS to use the intial
environment, rather than the current environment, introduced a bug such
that variables which had been cleaned from the environment where still set
in the data store.
This patch changes things such that a list of approved environment
variables is saved after the environment is cleaned and only the variables
in this list are inherited in inheritFromOS.
CC: James Limbouris <james.limbouris@gmail.com>
CC: Chris Larson <clarson@kergoth.com>
(Bitbake rev: cb6c07054e8baf94614713ec257c643b22266d75)
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Turn the reparseFiles logic into a command to reset the cooker's state
machine and a noop which triggers a cache rebuild.
The resetCooker command resets the cookers state machine such that a cache
update will be triggered by any async command which requires the cache.
The reparseFiles command remains as a noop async command that has the
needcache property set to True so that when called it ensures the cache is
built.
Patch from Richard with the addition of removing the force parameter from
the updateCache method.
CC: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: a98f698fe9f38310024013e58475e6d1447ee154)
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using the -e and -b options together an expection was occuring.
This was due to incorrect initialisation and this patch adds in the
correct initialisation calls.
(Bitbake rev: e7ade6dcd646b478df79f968934a0edcb63254f7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the requested configuration file is found on disk check the
against the configuration files in __depends/__base_depends to ensure the
file was parsed before emitting the ConfigFilePathFound event.
If the requested file wasn't parsed just return (and don't emit).
Fixes [YOCTO #1246]
(Bitbake rev: 705d14d1e1108e0544c7eab827f1242f0839add9)
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The include parameter to _parse() tells the parser whether to include the
file in __base_depends, the contents of which have their mtime tested to
invalidate the cache when they have changed.
As I understand it we would want all of the configuration files that are
currently handled with the _parse() method to invalidate the cache to set the
default value of the include parameter to True.
(Bitbake rev: 5cacdc4f1641eda1b5707c96f7c40924a9db6174)
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
"Nothing PROVIDES" errors often come up when a recipe has been skipped
for some reason, and therefore it is useful to print out that reason
information when showing the error so that the user understands why the
error has occurred.
Given that we already feed the reason information into the skiplist for
various situations (COMMERCIAL_LICENSE, COMPATIBLE_MACHINE etc.) this
should now output a useful error message for skipped recipes.
Fixes [YOCTO #846], [YOCTO #1127]
(Bitbake rev: 6765218430e31c165888f26fbc75023c89a6eab2)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton
Kang Kai
Richard Purdie
Dongxiao Xu
Lianhao Lu
Shane Wang
Caleb Crome
Christopher Larson
Christopher Larson
Robert Yang
Joshua Lock