We already export http,ftp,https proxy environment variables. Some environments,
GNOME for instance, place the socks proxy in ALL_PROXY and all_proxy. Export it
as well.
Signed-off-by: Darren Hart <dvhart@linux.intel.com>
We catch any exception raised by the fetchers go() method and attempt to work
around it by trying any (post) mirrors which are configured. However, should
the mirrors fail the exception is lost and the fetch is assumed to have
completed successfully.
Instead, save the exception and if the local file does not exist after trying
the mirrors re-raise the exception.
Fixes [BUGID #362]
Signed-off-by: Joshua Lock <josh@linux.intel.com>
[BUGID #375]
When configuring for the log file output, we generate a symlink to the
target filename. This link uses the same file naming, but without the
active pid, making it easier to see which log file is the last one
generated.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Pass task has informaiton to work processes, allowing full manipulation of
the hash data in the task context allowing checksums to be usable.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
With "no_proxy" exported to bitbake, both internal and externel file mirror can
be used. "https_proxy" enable fetching "https://" file through proxy.
Signed-off-by: Zhai Edwin <edwin.zhai@intel.com>
It was possible for bitbake-runtime to be run against a semi-installed
python-native resulting in tracebacks with ImportError's.
To prevent this we stash the initial PATH in the BBConfiguration when bitbake
is started and then set this in the env when launching bitbake-runtask through
subprocesses Popen() call.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Add a new method, try_premirror, to the Fetch object which checks to see
whether the file needs to be fetched from a premirror.
Override this in the Git fetcher to only require a pre-mirror fetch when the
clone directory does not exist.
Fixes [BUGID 290]
Signed-off-by: Joshua Lock <josh@linux.intel.com>
When setting localpath in the SRC_URI parameters the basename should be set
explicitly as the localpath() method is not called.
Fixes [BUGID #81]
Signed-off-by: Joshua Lock <josh@linux.intel.com>
If we fetch a tarball from a mirror it's entirely possible that the
mirror will have been from before the required tag/branch/etc was included
in the repository. To that end use forcefetch() as a way of testing whether
the repository is up to date and if not fetch updates.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
The git fetcher should force a fetch if the required tag is not present in
the local clone, or if the fullclone parameter is set.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
When pulling from a premirror we would prefer a local tarball even when the
caller had specified the forcefetch parameter.
Add an extra parameter 'force' to try_mirrors, defaulting to False. If set
the mirrors will be tested even if the file exists locally.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
For a given input to this code, the output doesn't change to implement a persistent
cache of the data to speed up parsing.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
These functions allow generation of dependency data between funcitons and
variables allowing moves to be made towards generating checksums and allowing
use of the dependency information in other parts of bitbake.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This commit is derived from Chris Larson's checksum work, turned into a
standalone piece of code for parsing python and shell functions.
The deindent code has been replaced with code to work around indentation
for speed. The original NodeVisitor in the ast was replaced with a faster
class walk call.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Add a new extend_deps function to more easily merge two dependency lists.
This avoids adding duplicates, unless the value of the dependency is
different.
Signed-off-by: Mark Hatle <mhatle@windriver.com>
It appears the timeout sometimes has no effect and we see database access failures. Combat
this by wrapping the execute function in all cases and retrying manually ourselves.
Thanks to Kevin Tian for help debugging this.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
bitbake: scenequeue: Skip setscene if the underlying task already ran
bitbake/setscene: Make sure uneeded dependencies are removed recursively
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
the 'fn' argument of them is not used, should be removed anytime:
sed -i -e
'/^def.*fn/s/,[[:space:]]*fn[[:space:]]*=[[:space:]]*None[[:space:]]*)/)/g'
lib/bb/msg.py
(Bitbake rev: 1cb72e371322c271ee7f2d008c6f7899fb38b4fd)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Searches the module (bb.runqueue) for any new style classes which are
instances of RunQueueScheduler, and uses the one whose 'name' attribute
matches the value of BB_SCHEDULER.
(Bitbake rev: 6497cedf9cfc03201250af816995dd2bd85c36ef)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
to make python3 happy
(Bitbake rev: c82926ccdd4ec4e3ad6e78a381dacb96adf9b409)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Likely cause is, of course, typo in the pattern or incorrect BBFILES, so we
should warn the user about this.
(Bitbake rev: b781317b5006bc047a59e7fa3c93344115e78ccb)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This failure is one the *user* needs to see, as it may have to do with their
setup, but where they don't need to see a traceback and all, since they're not
developers, so I think SystemExit is appropriate here.
(Bitbake rev: e3cc9bc7b6ef1a2eca78235790a051329bceb9a4)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This is necessary, as there has not yet been a release with the bb.utils
versions in place. We can't show them a deprecation warning when they can't
safely change it yet. Show a PendingDeprecationWarning until 1.10 releases
and OE requires it, then we can move to DeprecationWarning.
(Bitbake rev: 43700fa1ca1d2635de4ed46110057cb164ee82d1)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Fixes proxy support to honor standard proxy environment variables.
* Quote environment variables
(Bitbake rev: f84f382f340d6db15b9e5afb8c7c93969249a958)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
time.strftime() defaults to using time.localtime(), use that instead
of spawning "sh -c 'date +<FMT>'"
(Bitbake rev: ec1f2773b4811b3beb52710120bb5a9b215c174e)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
previously we called chdir() twice for every target, this patch reduces
the amount of chdir() calls via openembedded master from some 16000 to
4.
(Bitbake rev: fa45f5625e13a82bec70d5f10815f52fbe705166)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Store database connection to persistent database in fetcher.
(Bitbake rev: 8a6876752b90efd81d92f0947bfc9527d8260969)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
to make updating depends easier/more intuitive/eventually faster
(Bitbake rev: f7c69462b8ba726861898817cc5b13174c78e35a)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
(Bitbake rev: c0cf85beda4cf8748fd270c037442cde7b98146b)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
(Bitbake rev: 09fe0d403c75229931c10eabc7eecc0dc60058fe)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
The root cause is, I was testing the use of renameVar() from finalize, in
order to get flags copied over when an override is applied, but renameVar
removes the original, whereas the old code did not do so. Going back to the
old method, will revisit the override/flags later on.
(Bitbake rev: 2f7c498abcf675e5b8de197d8056a0581670c2bd)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This ensures the check is run for anyone using the package, including
setup.py, though it's sad that it's necessary, since all it really wants is
__version__.
Ideally, we'd avoid the version check entirely in favor of checking for
functionality, but that's rather difficult with language constructs like
context managers, so this'll do for now.
(Bitbake rev: f936f1c0edf70860ea0f39ec1a500ba43b56f289)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
See the comment in the code for details.
(Bitbake rev: 849dbd63244cbc4eaca0f1beedbb67baca024629)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This way we can fully utilize bblayers, you can do everything in bblayers.conf
and avoid setting any environment variables at all.
(Bitbake rev: 5def1c8c31432968349f9b29d6333d7962260a8b)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
types.IntType -> int
types.StringType -> basestring
...
Also moves our ImmutableTypes tuple into our own namespace.
(Bitbake rev: 83674a3a5564ecb1f9d2c9b2d5b1eeb3c31272ab)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
As an example, this displays a deprecation warning for the use of
"bb.encodeurl" when you should be using "bb.fetch.encodeurl". It includes a
convenience function for this purpose. It should be of use when moving
objects between modules permanently, changing the API the user sees.
(Bitbake rev: 78f56049ba863b2e585b89db12b32697eb879bbc)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
We should add back in some proper unit testing. Doctest is rather limited,
and we haven't actually made use of it since the original implementation of
the datastore when the project started, as far as I'm aware.
(Bitbake rev: 3a11c2807972bbbddffde2fa67fc380d159da467)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This lets you do -l Default on the commandline to bump the debug level of
messages with no domain, without bumping the level of everything else.
(Bitbake rev: bef33dfc36713d8c93e38e63ddac2e4f9858e787)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Also fixes some bb.msg references from within bb.msg.
(Bitbake rev: db95af590f742c8186e84046ad9704fae1733720)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Apparently the finalise spelling is becoming less common in British English.
(Bitbake rev: 47449b2fc433e5725839ca4f7e9bca931a475838)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
We should add back in some proper unit testing. Doctest is rather limited,
and we haven't actually made use of it since the original implementation of
the datastore when the project started, as far as I'm aware.
(Bitbake rev: 3a11c2807972bbbddffde2fa67fc380d159da467)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
When BBCLASSEXTEND is set, '-b' builds usually failed with messages
like
| ERROR: Parsing error data_fn virtual:native:<recipe>.bb and fn
<recipe>.bb don't match
|
| File ".../bb/providers.py", line 47, in sortPriorities
| priority = dataCache.bbfile_priority[f]
| KeyError: 'virtual:native:<recipe>.bb'
This patch fixes it and allows to specify the alternative class in a
way like
| ./bitbake -b virtual:native:<recipe>.bb
This patch was written to be so minimal as possible; variables should be
probably renamed to reflect their new meaning.
(Bitbake rev: f1c7fe9fc12161ceb3fe201cde370b929b208729)
Signed-off-by: Enrico Scholz <enrico.scholz@informatik.tu-chemnitz.de>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This is just for compatibility. We may drop it in the future, or rewrite it,
as it's not particularly pythonic.
(Bitbake rev: c4e31d7fe1d15a1e3ef2a453e7c7812d403d22cb)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Cache only uses the config metadata from the cooker, no need for the cooker itself.
(Bitbake rev: bf58d43444642d31293a341ce72292003ca5c162)
Signed-off-by: Chris Larson <clarson@kergoth.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
The methodpool, ${@} expansions, anonymous python functions, event handlers
now all run with the same global context, ensuring a consistent environment
for them. Added a bb.utils.better_eval function which does an eval() with the
same globals as better_exec.
(Bitbake rev: 424d7e267b009cc19b8503eadab782736d9597d0)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
- Use a single dictionary for the context, both global & local, since for some
reason it chokes wanting a global "d" rather than a local in the metadata.
- First compile the string into a code object before running eval, so we can
include the variable name in an evaluation error.
(Bitbake rev: 49534d928a37e0804ca84eed186cd22363023b2e)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
In order to move the environment cleaning, which wants to log a
message, before cooker creation, the logging facility initialization
needs to happen earlier. So, it is now in init_logger in utils.py and
called from bitbake before the creation of the data object.
It also seems more correct to initialize a global facility like this
from a more global context than the creation of an object, of which
there could theoretically be many.
Signed-off-by: Jeff Dike <jdike@linux.intel.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Without this patch decoding a url of this kind file://dir/filename gives
path=/filename host=dir.
With the patch it decodes as path=/dir/filename host=""
Probably nobody stumbled on this issue yet because nobody used
file:// urls with directory names in the path.
Signed-off-by: Nitin A Kamble <nitin.a.kamble@intel.com>
The following utility functions were removed. Note that in this particular
case we're bypassing a proper deprecation process, as these functions are
clearly not utilized, are obvious remnants of old ways of doing things, and
some of which do not even function properly.
- tokenize
- evaluate
- flatten
- relparse
- ververify
- isjustname
- isspecific
- catpkgsplit
- pkgsplit
- pkgcmp
- dep_parenreduce
- dep_opconvert
(Bitbake rev: 62983ad9b151ee8d51e8cf9a31c736c7813edf16)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This implements a feature similar to BBCLASSEXTEND, but for generating
multiple versions of a given recipe. For example: BBVERSIONS = "1.0 2.0 git".
In addition to the above, one can utilize [a-b] style patterns, and can have a
:<basever> postfix, which allows you to essentially name the range of
versions. Both the current version and the basever end up in OVERRIDES, and
the basever gets placed into the BPV variable. The default BPV, if none is
specified, is the original PV of the recipe, before bbversions processing.
In this way, you can do things like:
BBVERSIONS = "1.0.[0-6]:1.0.0+
1.0.[7-9]:1.0.7+"
SRC_URI_append_1.0.7+ = "file://some_extra_patch.patch;patch=1"
Or you can create a recipe per range, and name the recipe file as such: nano_1.0.7+.bb.
(Bitbake rev: 4ee9a56e16f1eb3c1649eaa3127b09ab0e93d1ec)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
When not expanding PREMIRRORS, the functions fails/does not work correctly
when PREMIRRORS is not a plain string (e.g. contains ${...} or a ${@...}
statements).
(Bitbake rev: d612d22b073f68b8cf1bb7344e0487820040d80d)
Signed-off-by: Enrico Scholz <enrico.scholz@informatik.tu-chemnitz.de>
Signed-off-by: Chris Larson <clarson@kergoth.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
(Bitbake rev: b2486ec57c6a7adf09d0960fdf6727881b324d2f)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Chris Larson <clarson@kergoth.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This prevents a misleading backtrace:
ERROR: no files to build.
Command execution failed: Traceback (most recent call last):
File ".../bitbake/build/lib/bb/command.py", line 83, in runAsyncCommand
self.cooker.updateCache()
File ".../bitbake/build/lib/bb/cooker.py", line 779, in updateCache
if not self.parser.parse_next():
File ".../bitbake/build/lib/bb/cooker.py", line 969, in parse_next
cooker.bb_cache.sync()
UnboundLocalError: local variable 'cooker' referenced before assignment
(Bitbake rev: 060ef3d957615b7eb1209dc0d01ebeb53f8c4edc)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Chris Larson <clarson@kergoth.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
??= is a lazy, conditional assignment. Whereas a ?= immediately assigns to
the variable if the variable has not yet been set, ??= does not apply the
default assignment until the end of the parse. As a result, the final ??= for
a given variable is used, as opposed to the first as in ?=.
Note that the initial implementation relies upon finalise() to apply the
defaults, so a "bitbake -e" without specifying a recipe will not show the
defaults as set by ??=. Moving application of the default into getVar adds
too large a performance hit. We may want to revisit this later.
(Bitbake rev: 74f50fbca194c9c72bd2a540f4b9de458cb08e2d)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
dict objects provide an __iter__ method for the iteration which gives you the
keys, so calling keys directly is unnecessary, and isn't really a best
practice. The only time you really need to call the keys is if there's a
danger of the dict changing out from underneith you, either due to external
forces or due to modification of the iterable in the loop. Iterations over
os.environ are apparently subject to such changes, so they must continue to
use keys().
As an aside, also switches a couple spots to using sorted() rather than
creating a temporary list with keys() and sorting that.
(Bitbake rev: 5b6ccb16c6e71e23dac6920cd2df994d67c2587b)
Signed-off-by: Chris Larson <clarson@mvista.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
..to make copy and paste of the logfile easier.
(Bitbake rev: 446cc0cebd4daff7f849717f4cb89ac1b4c6b755)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Chris Larson <clarson@kergoth.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
(Bitbake rev: f68406e864c9837feb56cbec993b620481445cc2)
Signed-off-by: Tom Rini <tom_rini@mentor.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This ensures that an anonymous python function is able to manipulate the
BBCLASSEXTEND contents, and, therefore, amend.inc files are able to add to it.
(Bitbake rev: c7d038d404afaf4ce3735af5134163759da6f6ef)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Better to error as early as possible rather than experience strange behavior
resulting from the use of the largely useless stock bitbake.conf/base.bbclass.
(Bitbake rev: 641e6cf3ec3ab4d26929cf4d2a3704ff07eed4d6)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
The Git fetcher currently hardwires the git command to "git". Allow the
path and any additional wrappers to the Git command to be provided via
FETCHCMD functionality, as with some of the other fetchers.
If FETCHCMD_git is not define in bitbake.conf, the fetcher defaults to "git".
(Bitbake rev: f3afb79ecac30d973a3c62ff6baf28d8b7388a24)
Signed-off-by: Malcolm Crossley <malcolm.crossley@ge.com>
Signed-off-by: Martyn Welch <martyn.welch@ge.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This breaks preferred providers functionality
This reverts commit ee9afccf33b220a21b74fab279925eeb4771249b.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Do not attempt to open the file in the resolve_file method
(a lot like bb.which... maybe bb.which can be used). This way
we don't need to open/close a file which we have already parsed.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Have a growing dict with .inc and .bbclass'es. This avoids to reparse
files we have already seen.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Our parser is shit but instead to replace it now we will see
how long we can drive the wave by caching parsed files. This
will not go through the feeder again but we can just reevaluate
the StatementGroup.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Evaluate the statements after having parsed one file. This is
referred to as "entwirren" and we can remove the direct evaluation
and postpone a bit, in the future we can use a cached copy instead
of parsing the original.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
First set of ConfHandling with AST nodes. The include can
use a speed up and things might need to be migrated... into
this class.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
When parsing we will collect a number of statements
that can be evaluated...The plan is to be evaluate
things twice (old+new) and then compare the result,
it should be the same.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
