Various misc changes to convert bitbake to python3 which don't warrant
separation into separate commits.
(Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Under python the type conversions can mean there are float values
used for triggering the parse progress events which then fails.
Add an explict int() conversion to ensure the parse events are
generated under python3.
(Bitbake rev: 138329c58e92744c56aae3ab70ceeef09613250c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We don't need to pass the DATABASE_URL around and read it back if we
setup the django framework in the correct way.
We make the default sqlite database path a full path so that the
database isn't being assumed to be in CWD.
Also add some more useful comments on the database settings.
This is preparation work to migrate the build tests and be able to
trigger builds on differently configured databases.
(Bitbake rev: 973c740404ca6a09feea250d3433075995067fe0)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use open() instead of file() and close files when finished with them.
(Bitbake rev: 033c5a16ff19781ed793c2d97d285884017a2a4e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously we only tracked the flags (minus excluded) of variables we depend
on, but not the flags we use explicitly.
(Bitbake rev: bdeb3dcd7c92e62a7c079e7b27048c4114f24a3a)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This variable is a regex-escaped version of LAYERDIR, for safer use in
BBFILE_PATTERN, so as to avoid issues with regex special characters in the
layer path.
[YOCTO #8402]
(Bitbake rev: 72900522778b6ff08b135bf8bb97dff3f1a20bd9)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It's useful to see tracebacks for ExpansionErrors, but only if we skip the
leading bitbake-internal elements, otherwise we see elements of the expansion
process.
As one example:
Before:
ERROR: ExpansionError during parsing /scratch/yocto-new/external-as-needed/poky/meta/recipes-core/glibc/glibc-locale_2.23.bb: Failure expanding variable PV[:=], expression was ${@get_external_libc_version(d)} which triggered exception AttributeError: 'module' object has no attribute 'external'
After:
ERROR: ExpansionError during parsing /scratch/yocto-new/external-as-needed/poky/meta/recipes-core/glibc/glibc-locale_2.23.bb
Traceback (most recent call last):
File "PV[:=]", line 1, in <module>
File "/scratch/yocto-new/external-as-needed/meta-sourcery/recipes-external/glibc/glibc-external-version.inc", line 3, in get_external_libc_version(d=<bb.data_smart.DataSmart
object at 0x7f05d2566950>):
sopattern = os.path.join(d.getVar('base_libdir', True), 'libc-*.so')
> found_paths = oe.external.find_sysroot_files([sopattern], d)
if found_paths:
ExpansionError: Failure expanding variable PV[:=], expression was ${@get_external_libc_version(d)} which triggered exception AttributeError: 'module' object has no attribute 'external'
(Bitbake rev: 7ff5b9eed82b7f4fd138fc6d746a0b79efbea98a)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to flush the footer removal, else it may not be outputted until
the buffer is flushed as part of StreamHandler and this would lead to
it removing the ERROR output just printed which is extremely confusing.
Also ensure the footer is cleared before printing a summary as in
some cases it wasn't being removed, also leading to user confusion.
(Bitbake rev: 0e030c4d074c41859608dab5f3ad26b05f56b306)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The signature data file comparison functions are meant to be able to
handle data files containing just the base hash data. This had regressed
in some places so add fixes to allow these comparisons to be made. The
runtime components in the data files are optional.
(Bitbake rev: 2a6659fd748e255a02c2f9d047829d6edfe65317)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
knotty captures two levels of keyboard interrupt: a single interrupt
or two interrupts in a row. These then trigger stateShutdown
and stateForceShutdown respectively.
toasterui doesn't have an equivalent way of capturing interrupts and
using them to shut down bitbake. Now that we are no longer using
knotty + XMLRPCServer for our command line builds (since switching to
per-project build directories), we see some odd side effects of this,
such as builds continuing after they have been interrupted on the
command line.
Bring toasterui in line with knotty (copy-paste most of the code
in knotty.py which deals with interrupts) so that a keyboard
interrupt actually shuts down the bitbake server (if not in
observe only mode).
Additionally use the cancel_cli_build() method to set the Build
status to CANCELLED in Toaster's db when we get keyboard interrupts.
This means that builds interrupted on the command line show as
cancelled (same as if they'd been cancelled from the Toaster UI),
as specified in the UI designs.
[YOCTO #8515]
(Bitbake rev: d39d2edca95900da433074ee95a192d7bfe7090d)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This will be used from toasterui to cancel the current command-line
build when a keyboard interrupt is captured.
[YOCTO #8515]
(Bitbake rev: 1486c770327b53bb5e04baa5f3ea26d8154aed63)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if the server dies, its possible that log messages are never
displayed which is particularly problematic if one of those messages
is the exception and backtrace the server died with.
Rather than having the event queue exit as soon as the server disappears,
we should pop events from the queue until its empty before exiting.
This patch tweaks that code so that even if the server is dead and we're
going to exit, we return any events left in the pipe. This makes
debugging certain failures much easier.
(Bitbake rev: 29f6ade68fb2b506a23a7eb3a00cdcffa291b362)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now the event is sent unconditionally we can drop this feature
as its no longer needed.
(Bitbake rev: 473deeb0fc6065693e1fcfcbb8b79753103db537)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The data included in the event is useful for implementing a pre-build
check that warns about unexpected components, for example because of
an incorrect configuration or changed dependencies.
Such a check can be done in a .bbclass that gets inherited
globally. But in contrast to a UI, such a class cannot request that
the event shall be emitted, and thus the event has to be emitted
whether there is a consumer or not.
This was done conditionally earlier out of concerns about the
performance impact. But now events are handled more efficiently, so
that concern no longer seems valid: in some simple testing (admittedly
on a fast build workstation), the two lines (generating the data and
emitting the event with it) only took about 0.05 seconds (measured
with timeit). That was for a build with roughly 500 recipes (from
pn-buildlist aka depgraph['pn']), triggered via the command line. That
was even with a consumer of the data active and doing some work, so it
should be even faster when there is no consumer.
(Bitbake rev: 5ddaf5b7ed1001d2dd3f67e7a6d704afa85479d2)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the cooker fails to start, ensure a correct exception is displayed to the
user. After handling any queued events simply re-raise the original exception
else the output can be unclear.
(Bitbake rev: 9a4db1aa608c17d31bf5ea1cab5a99beb565dd83)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The string formatting wasn't correct and we should exit if we hit
errors here similar to the other exception handlers.
(Bitbake rev: b90a16408a5c45ce5312384f278e19d09f8dda4d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
python3 can't cope with the previous approach we were using to pass
exceptions through the RPC. Avoid this by creating a formatted exception
on the sender side.
(Bitbake rev: d7db75020ed727677afbad07a90fb3eac0bf2c45)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this, the dict can reorder causing sanity test failures.
(Bitbake rev: ca8c91acc9396385834b266d4e8b84d917e5e298)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This avoids a deprecation warning in python 3.
(Bitbake rev: bf1a92d0c002d73e8a34472dced1343dc4a4251a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ensure we pass the string parameter correctly.
(Bitbake rev: 7ed82bd1fe7bdd93b0614119c42eb218dc5d83e6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Under python 3, if we spawn python processes, we need to have a UTF-8
locale, else python's file access methods will use ascii. You can't
change that mode once the interpreter is started so we have to ensure
a locale is set. Ideally we'd use C.UTF-8 since OE already forces the
C locale but not all distros support that and we need to set something.
Was tempted to choose en_GB so colour gets spelt correctly :).
This is in some ways pretty nasty, forcing it into the environment
everywhere however we only have a limited number of ways of making
everything work correctly and this beats having to add utf-8 encoding
to every file access command.
A similar change will be needed to bitbake.conf in OE.
(Bitbake rev: 8902c29638411d312e6fc4a197707e5742652e15)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some tweaks to use python 3 syntax in a python 2 compatible way.
(Bitbake rev: 322949c77dbaa4db01b5a43d85b39a2af67ba7b2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we don't close the console log file handle, python prints a warning
about unclosed file handles upon exit which is annoying.
(Bitbake rev: 624dd92952b2fc736fd86abe5f2390b87b3a7dd3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
python3 cares more about invalid type comparisons. Add break statements
and better tests to make the code paths clearer and avoid type issues
in python3. No code functionality change.
(Bitbake rev: 2c39ebdd2762d027f007a6a769fdf023cdf3da2b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We shouldn't try and use fakeworker when performing a dry_run. This
makes the core match the other fakeworker execution points.
(Bitbake rev: 49bea821a2edad5e19c3a566d1a80c23718dede9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we pass data into explode_dep_versions2(), we need to result to be
able to be processed in a deterministic way so that we end up with
consistent hash values. This means we need an ordered structure rather
than an unordered one.
To do this, return an OrderedDict() rather than a dict().
(Bitbake rev: 0737e003ca549d08a7dfe13452ae982f2e11fecd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The preferred form is assertEqual, assertEquals is deprecated and
not present in python v3.
This is v2.7 safe.
(Bitbake rev: b60261bf8ade14aca31238b50c243c01adcabc59)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
python deprecated logger.warn() in favour of logger.warning(). This is only
used in bitbake code so we may as well just translate everything to avoid
warnings under python 3. Its safe for python 2.7.
(Bitbake rev: 676a5f592e8507e81b8f748d58acfea7572f8796)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This avoids a common issue where PACKAGECONFIG is emitted as a function in
bitbake -e when the 'python' flag exists. It isn't a python function unless
both 'func' and 'python' are set. This aligns with the behavior of
emit_func_python.
(Bitbake rev: c5e45063cb3ae17bbe3304ea5e712bd76e686c4a)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This lets us avoid treating the module like an object, so no globals are
needed, if one chooses to do so.
(Bitbake rev: 71bfd5beb0d0ed88c7c14bbfd5ca1a1b56122bc1)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than hardcoding .py, use python's knowledge of its file extensions.
(Bitbake rev: 09f838dbaefdaedc01a1f4818ed38280b38db744)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Imported as of oe-core 184a256.
(Bitbake rev: 99db61bf816d9c735032caa762aae8e6a0803402)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It seems the frozenset constructor in pypy runs len(), so we can't pass the
DataSmart instance directly to it, instead pass the iterator. Fixes pypy
support.
(Bitbake rev: b492836e08745e04bd9ba2fb0b56a680a5fdce79)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BB_ORIGENV value on the datastore can be NoneType thus raising an AttributeError
exception when calling the getVar method. To avoid this, a check is done before
accesing it.
[YOCTO #9567]
(Bitbake rev: f368f5ae64a1681873f3d81f3cb8fb38650367b0)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Wrapped long lines to fix "Line too long" pylint warnings.
(Bitbake rev: e329a932e14d002a561245b5026f974897f64598)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed pylint warnings:
No space allowed around keyword argument assignment
No space allowed after bracket
No space allowed before bracket
(Bitbake rev: c39770239f7b61217501782b9c5e9d3211355d42)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Environment variables BBSERVER, BBTOKEN and BBEVENTLOG silently
overwrite bitbake command line arguments. This is confusing and
can cause issues that are difficult to debug. It's better to use
them as default values instead.
Used environment variables BBSERVER, BBTOKEN and BBEVENTLOG to set
default values for command line arguments.
Changed setting default value of --ui command line argument from
BITBAKE_UI to look similar way.
(Bitbake rev: 87040be4ff54cd460961318224deef8f2ea4c85a)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake uses set of environment variables to set command line
options, e.g. seeting BBTOKEN variable has the same effect
as using --token command line option.
Added new environment variables BBPRECONF and BBPOSTCONF that
are equivalents of --read and --postread command line options.
They can be used by high level scripts to append or prepend
configuration files to conf/local.conf
[YOCTO #9235]
(Bitbake rev: bf604ec1ca4eb4d0b22bcc72249963e6d7445f34)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some users may want to use authenticated SSH connections with credentials stored
in a keyring, such as gnome-keyring. These typically need a DBus session bus
connection, so pass DBUS_SESSION_BUS_ADDRESS into the fetcher environment.
To avoid the user needing to set it in their local.conf (which wouldn't be
usable) or adding it to the environment-cleansing whitelist (which would
potentially impact builds) allow the variables being passed to the fetchers to
come from the data store (first) or the original environment (second).
(Bitbake rev: 20ad1ea87712d042bd5d89ce1957793f7ff71da0)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed typo that caused incorrect processing of BBEVENTLOG
environment variable. Even if variable is set it was ignored
by bitbake.
(Bitbake rev: 2705b5f59aef4a070e2df2752d27bd04ea747057)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie
Michael Wood
Christopher Larson
Elliot Smith
Patrick Ohly
Leonardo Sandoval
Ed Bartosh
Ross Burton