Now you get the bb logger output for failed tests. This helps debugging
problems. Also, all stdout/stderr data for successful tests is silenced
which makes for less cluttered console output.
(Bitbake rev: ea19972a16f7639f944823d1d8a7728105460136)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Set this env variable to 'yes' to preserve temporary directories used by
the fetcher tests. Useful for debugging tests.
(Bitbake rev: 04132b261df9def3a0cff14c93c29b26ff906e8b)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This simplifies the script, and, gives new features. It is now possible
to run single test functions, for example. This is nice when writing new
test cases.
(Bitbake rev: 8c513580b9406b031674f799117eae7410f8e01c)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds the notion of supporting multiple configurations within
a single build. To enable it, set a line in local.conf like:
BBMULTICONFIG = "configA configB configC"
This would tell bitbake that before it parses the base configuration,
it should load conf/configA.conf and so on for each different
configuration. These would contain lines like:
MACHINE = "A"
or other variables which can be set which can be built in the same
build directory (or change TMPDIR not to conflict).
One downside I've already discovered is that if we want to inherit this
file right at the start of parsing, the only place you can put the
configurations is in "cwd", since BBPATH isn't constructed until the
layers are parsed and therefore using it as a preconf file isn't
possible unless its located there.
Execution of these targets takes the form "bitbake
multiconfig:configA:core-image-minimal core-image-sato" so similar to
our virtclass approach for native/nativesdk/multilib using BBCLASSEXTEND.
Implementation wise, the implication is that instead of tasks being
uniquely referenced with "recipename/fn:task" it now needs to be
"configuration:recipename:task".
We already started using "virtual" filenames for recipes when we
implemented BBCLASSEXTEND and this patch adds a new prefix to
these, "multiconfig:<configname>:" and hence avoid changes to a large
part of the codebase thanks to this. databuilder has an internal array
of data stores and uses the right one depending on the supplied virtual
filename.
That trick allows us to use the existing parsing code including the
multithreading mostly unchanged as well as most of the cache code.
For recipecache, we end up with a dict of these accessed by
multiconfig (mc). taskdata and runqueue can only cope with one recipecache
so for taskdata, we pass in each recipecache and have it compute the result
and end up with an array of taskdatas. We can only have one runqueue so there
extensive changes there.
This initial implementation has some drawbacks:
a) There are no inter-multi-configuration dependencies as yet
b) There are no sstate optimisations. This means if the build uses the
same object twice in say two different TMPDIRs, it will either load from
an existing sstate cache at the start or build it twice. We can then in
due course look at ways in which it would only build it once and then
reuse it. This will likely need significant changes to the way sstate
currently works to make that possible.
(Bitbake rev: 5287991691578825c847bac2368e9b51c0ede3f0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than passing in a datastore to build on top of, use the data builder
object in the cache and base the parsed recipe from this. This turns
things into proper objects building from one another rather than messy
mixes of static and class functions.
This sets things up so we can support parsing and building multiple
configurations.
(Bitbake rev: fef18b445c0cb6b266cd939b9c78d7cbce38663f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This has been replaced using django's inbuilt loaddata.
Django command documented at:
https://docs.djangoproject.com/en/1.8/ref/django-admin/#django-admin-loaddata
(Bitbake rev: 3edd33aff2b9eef82090dc4f9b5461f901aec8be)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Call django's inbuilt loaddata command to load the appropriate fixtures.
We also attempt to load a fixture called "custom" and fail silently if
we don't have one. This is where initial customisations can be done to
load particular settings or data into Toaster (for example layers or
default values for variables)
Make sure the value for TEMPLATECONF is available to checksettings so
that we can have a go a working out which default data to load.
(Bitbake rev: 7d14ca8cbabbb893e507a66e4cc6e3e77c1e8c84)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the user explicitly passes in "--help" then it should return 0. This
is the convention follow by the typical application. This allows the
user to check for options without triggering an error.
(Bitbake rev: 1d5102fe6c932dad1c2b975385e10a33a91ba1a4)
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rewritten toaster-eventreplay to make code working as expected,
more compact and readable.
[YOCTO #9585]
(Bitbake rev: 45370a860b24a761d1b6e08ba752079cc45f54da)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
class MockConfigParameters has only one attribute and only __init__
method. Replacing it with namedtuple makes code less nested and more
readable.
[YOCTO #9585]
(Bitbake rev: 5d4df14b0d38f6c89ca16de6dada58b4bb015d71)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Event objects are represented by base64-encoded strings in
the event file and can't be loaded by existing eventreplay code.
Fixed the code of loading events from file by decoding base64 strings
into the binary form and loading them with pickle.load.
[YOCTO #9585]
(Bitbake rev: a55c280c167f84caed6518119246e5a55f56cfd4)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Stored event mask list as self.eventmask for future use.
Fixed Exception: Command setEventMask not implemented.
[YOCTO #9585]
(Bitbake rev: 2e09074c70e89402de8f883dd402cd729118fc7e)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed AttributeError: 'MockConnection' object has no attribute
'getEventHandle'
[YOCTO #9585]
(Bitbake rev: 6deae30480a4288da0c8b3529e61f3495f260f24)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Worker needs input stream in binary mode as it reads binary content
from it. Current code does it by detaching a buffer from sys.stdin
and assigning it back to sys.stdin.
Detached buffer is io.BufferedReader in binary mode. This operation
is implicit as its purpose is not easily understandable from the code.
Replacing it with fdopen(sys.stdin.fileno(), 'rb') should make the
code more understandable.
Assigning the buffer to sys.stdin is not needed as worker doesn't
use sys.stdin. Moreover, it leads to difficult to debug issues down
the stack. For example, devpyshell doesn't work without reopening
sys.stdin in text mode. This is not needed anymore after this fix as
sys.stdin is not changed in worker code and remains in text mode.
(Bitbake rev: b26bcff4c4d72775f1def7e769015464953b955c)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Explicitly use python3 so that the modules for python3 are checked.
(Bitbake rev: e7951541c34c5561187110ba0ec69b9c45022747)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Explicitly used python3 as default python for oe builds
will continue to be python2.
[YOCTO #9584]
(Bitbake rev: fde5c962cb69a11b072d1f238c2371a5137d030d)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The upgrade to python3 is the final nail in the coffin for image-writer
and the goggle UI. Neither seem used or recieve patches and are based
on old versions of GTK+ so drop them, and the remaining crumbs support
pieces.
(Bitbake rev: ee7df1ca00c76f755057c157c093294efb9078d8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Disable the problematic gtk usage for use with pygtkcompat. The following
commit removes these tools/UIs entirely but we may as well leave this
piece in the history in case anyone does want a starting point for reusing
them.
(Bitbake rev: c53c7418d392452450352ca2175667dbdbd92401)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Various misc changes to convert bitbake to python3 which don't warrant
separation into separate commits.
(Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We don't need to pass the DATABASE_URL around and read it back if we
setup the django framework in the correct way.
We make the default sqlite database path a full path so that the
database isn't being assumed to be in CWD.
Also add some more useful comments on the database settings.
This is preparation work to migrate the build tests and be able to
trigger builds on differently configured databases.
(Bitbake rev: 973c740404ca6a09feea250d3433075995067fe0)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This uses bb.utils.load_plugins, based on the plugin handling in recipetool
and devtool in oe-core.
(Bitbake rev: 5e542df9b966a99b5a5b8aa7cf6100174aff54b2)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Used print() function instead of print statement
to make toaster script to work with both python 2 and python 3
[YOCTO #9584]
(Bitbake rev: 09d37ee51219edcd0be6fd24c82fce392533b39b)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The logging domain specified to bb.msg.fatal was invalid. Replace with
a logger.critical() call instead.
(Bitbake rev: 1ffd8737e065a3cd634c74cd67e634d785ea93a5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This updates the print "" syntax to print() and fixes some exception
handling syntax such that its compatible with python v2 and v3.
(Bitbake rev: 58304fcce9727fd89564436771356c033ecd22a3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
python deprecated logger.warn() in favour of logger.warning(). This is only
used in bitbake code so we may as well just translate everything to avoid
warnings under python 3. Its safe for python 2.7.
(Bitbake rev: 676a5f592e8507e81b8f748d58acfea7572f8796)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For the noweb command we don't need to check if a socket is in use or
not as we're not starting the django development server. We're just
setting up the environment and running the runbuilds scheduler.
(Bitbake rev: f62d9a3e86e384f928fc8ad077d7cf3a75d1591e)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow binding to a specified address and port to start the django
development server on.
Remove the assumption that you want to bind to 0.0.0.0 and set the
default to localhost.
Add some additional help text after Toaster webserver has started
(Bitbake rev: ef3c88a57313947feb5ef08c4fea6fd110cde1e7)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add help text for unrecognised or missing command for toaster script
Remove assumption that no command is 'start' as the default.
(Bitbake rev: 4c0b1b9c2af70838dbcf5a676ec679f9f0c488a5)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This variable will be used in localhost controller code
to determine correct bitbake location, so it has to be
exported.
(Bitbake rev: 429d47325aadb74d476e5b7f4738f2dfe26cde95)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Added 'INHERIT+="toaster buildhistory"' line to the conf/local conf
when Toaster starts. It should make commandline builds to provide
all required information to Toaster backend.
(Bitbake rev: 1271cf430087c66f87c46689b37b8a3538c35739)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Set BITBAKE_UI variable to 'toastergui' for command line builds
to use toasterui as a default ui module for bitbake.
(Bitbake rev: 9ad6393d30cb6196cf7c9a5adcf33febd724d294)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
noui command line option doesn't makes sense anymore as toaster doesn't
run bitbake. It should be safe to to remove it.
The purpose of this option was to skip running bitbake observer process.
This was never used before as it's not possible to run toaster build
without running observer.
(Bitbake rev: 7506719090e8bb39231cf389c4a5b47f1b37a01f)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>From now on toaster script will not run bitbake server.
It will be started by runbuilds and stopped after the build.
(Bitbake rev: 3fbd8534149e87c5a5d1bc1691711cfca05cafd1)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There appears to have been a lot of copy and pasting of the code
which prints tracebacks upon failure and limits the stack trace to
5 entries. This obscures the real error and is very confusing to the user
it look me an age to work out why some tracebacks weren't useful.
This patch removes the limit, making tracebacks much more useful for
debugging.
[YOCTO #9230]
(Bitbake rev: 5549748a200b5df259fc7352477ec59471b87b2f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This cleanup fixes a few issues:
1) Removes the superflous code to set toasterconf.json
- this isn't used and referenced meta-yocto
2) Changes exit to return so we don't surprise the user by exiting their shell
- this is necessary because it is being sourced
3) Removes the last references to the old TOASTER_MANAGED variable
- this is historical and no longer used.
4) Adds -t parameter to lsof
- This stops it from dying on odd filesystems and is much
faster since all we are using are processes anyway
5) Handles start and stop as params
- it was easy to confuse the script especially
if we were calling it with parameters.
- if start/stop isn't specified, it will still toggle
(Bitbake rev: 88fddbe80f56828026bf93560037af52b5dab628)
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This sets the scene for removing the default False for expansion from
getVarFlag. This would later allow True to become the expand default.
On the most part this is an automatic translation with:
sed -e 's:\(\.getVarFlag([^,()]*, [^,()]*\)):\1, False):g' -i `grep -ril getVar *`
There should be no functional change from this patch.
(Bitbake rev: 7c3b99c6a716095af3ffce0b15110e91fb49c913)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I before E, except after C...
(Bitbake rev: 14c9593265f7469cb8a205a46f845ac7491246df)
Signed-off-by: Phil Blundell <pb@pbcl.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This means that when you view the process tree, the processes
have meaningful names, aiding debugging:
$ pstree -p 30021
bash(30021)───KnottyUI(115579)───Cooker(115590)─┬─PRServ(115592)───{PRServ Handler}(115593)
├─Worker(115630)───bash:sleep(115631)───run.do_sleep.11(115633)───sleep(115634)
└─{ProcessEQueue}(115591)
$ pstree -p 30021
bash(30021)───KnottyUI(117319)───Cooker(117330)─┬─Cooker(117335)
├─PRServ(117332)───{PRServ Handler}(117333)
├─Parser-1:2(117336)
└─{ProcessEQueue}(117331)
Applies to parse threads, PR Server, cooker, the workers and execution
threads, working within the 16 character limit as best we can.
Needed to tweak the bitbake-worker magic values to tell the
workers apart.
(Bitbake rev: 539726a3b2202249a3f148d99e08909cb61902a5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Toaster script run bitbake with --postread conf/toaster.conf.
It turned out that variables set this way don't influence the
build. Changing --postread to --read makes bitbake to read
variables earlier and should fix this issue.
[YOCTO #8781]
(Bitbake rev: 76c9871740ef42ac35fdfdcb89a68478cca370cd)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sometimes Toaster connection to the bitbake server fails with
this error in the log:
ERROR: Could not connect to server 0.0.0.0:46572
: Could not register UI event handler
It happens more often on slow machines.
Increasing timeout after restarting bitbake should fix this issue.
[YOCTO #8776]
(Bitbake rev: b64a0adc1efd72aa4d46df3e043dd14543abae3d)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Removed nobrowser and brbe script parameters as both
are confusing and nobrowser is not used anywhere.
brbe parameter usage can only be justified if toaster
doesn't work properly and user has to manually connect
toaster to running bitbake server. Even in this scenario
it's very unlikely to achieve as toaster script is not
designed for this kind of usage.
(Bitbake rev: 0fd04ede3fda6894d97a5ef830b79dbbc9c6cf51)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Set default values of script parameters just before
they are parsed to increase readability.
(Bitbake rev: 627f0d6adcfe281ef0487bf15a35151f1ceff194)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Markus Lehtonen
Richard Purdie
Michael Wood
Randy Witt
Ed Bartosh
Christopher Larson
brian avery
Phil Blundell