bbmake hasn't existed for a while (a decade+) so we should
probably remove the references.
(Bitbake rev: 4170cf874850b950f31d2e36c895c110c0b096c3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These imports were from python 2.6 and earlier, 2.4 in some cases.
Drop them since we're all python3 now.
(Bitbake rev: 7ef12684e8647b006bf46cae695069d4bfece1cf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
At end of BBCooker needs to release fd's associated with pyinotify
watchers to avoid: Too many open files (EMFILE) error in different
scenarios like several instances of tinfoil.
[YOCTO #10873]
(Bitbake rev: ae6045b84978940c365c95c33d6996359c3e299d)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've been seeing oe-selftest failures under puzzling circumstances. It
turns out if you run oe-selftest on a machine with xmlrunner installed
and have the recent tinfoil2 changes, the launching of PR server crashes
leading to selftest hanging if using an autoloaded PR server.
The reason is that xmlrunner uses an io.StringIO object as stdout/stderr
instead of the usual io.TextIOWrapper and StringIO lacks a fileno() method.
We have to deal with both cases and in the python way, we try and then seek
forgivness if we see an AttributeError or UnSupportedOperation exception.
Unfortunately we have to deal with both cases as we may be performing a
traditiional double fork() from the commandline, or a larger python program.
[YOCTO #10866]
(Bitbake rev: 26243f04e3af652291d13e85c084057104fe155b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
People are struggling with multiconfig as the point the conf file
is injected into the data store is not what people expect. We can't
really use a post config since that is too late and we can't really
use a pre config file since that is too early. In OE terms, we need
something right around the local.conf point so it behaves in a similar
way.
A way to handle this is to set the new variable BB_CURRENT_MC to be the
currently selected multiconfig, then the metadata itself can choose
when to inject the approriate configuration.
(Bitbake rev: 518b9015c2be8d3894277a8e54890d6f04d656c0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fakeroot workers usually have dependencies that need to be ready before they
can be started. Starting them as a block therefore doesn't work as the
dependencies may or may not have been built. Therefore start the multiconfig
fakeworkers individually upon demand.
[YOCTO #10344]
(Bitbake rev: ab15486c636e451ed5a62e561980946887901c3b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In multiconfig, mutliple tasks can execute which share the same stamp file. These
must not execute in parallel, the idea is the first should execute, the subsequent
ones should see a valid stamp and get skipped.
The normal task execution code has stamps code to handle this, this adds similar
code to the setscene execute() function to handle the issue there too.
(Bitbake rev: 937acf267fa9e45f538695b2cf8aa83232a96240)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
setscene was being entirelu skipped for multiconfig variants as the tasks
were simply not being spotted. If the default config was also being built
it masked the problem. When this was fixed by using taskfn instead of fn
in lookups against dataCache, several other instances of this problem were
highlighted.
This goes through and corrects the setscene code to correclty use taskfn
instead of fn in the appropriate places meaning setscene tasks for multiconfig
now work correctly.
(Bitbake rev: cead85cf219c07b40aeb7c0d948c25f0429136db)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tasknames can now start with "multiconfig:" which broke the virtual: comparison code and
lead to unpredictable checksums with nativesdk recipes. This adds in handling for
the new additional prefix which unbreaks nativesdk builds when using multiconfig.
(Bitbake rev: 4da296c0c93391ec37f8b6e2245071657759771e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the UI shows X is building, possibly multiple times but doesn't
say which of the multilibs that might be. This adds a prefix to the task
name so the mulitconfig being built can be identified.
(Bitbake rev: d93ddd185dde53f50aea06ada094bb0d9b5ba4f0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The location of the fakeroot command and the various environmental values need
to be taken from the right multiconfig datastore, not the shared one. This
patch ensures the right one is used for cases like a split TMPDIR.
[YOCTO #10344]
(Bitbake rev: 813a7600bc26b7132ac36d1515571d321f566afb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Commit 873e33d0479e977520106b65d149ff1799195bf6 [fetch2/wget:
add Basic Auth from netrc to checkstatus()] causes "Fetcher failure
for URL: 'https://www.example.com/'. URL https://www.example.com/
doesn't work." on new builds when a user has a .netrc file but there
is no default and no matching host. The call to netrc.authenticators()
will return None in these cases and the attempted assignment to the
3-tuple will raise a TypeError exception. Add the TypeError to the
exceptions caught to get around this issue.
(Bitbake rev: c0c0af40ebddaf9dc99353c580a65d4c04295613)
Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2/wget uses urllib to check the status of the mirrors, wget will
use netrc to pass login and password information however checkstatus
will skip that.
This adds netrc login and password to checkstatus so both will work the
same.
(Bitbake rev: 873e33d0479e977520106b65d149ff1799195bf6)
Signed-off-by: Matthew McClintock <msm-oss@mcclintock.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For the purposes BB_SETSCENE_ENFORCE is designed for (in OE, it is used
by the installation process for the extensible SDK), we don't actually
need the whitelisted real tasks to execute - we just need to have them
in the dependency tree so that we get all of the setscene tasks they
depend on to run. Therefore we can actually dry-run those real tasks
i.e. they won't be run (and thus we won't waste a significant amount of
time doing so) and won't be stamped as having run either. We do already
have a dry-run mode in BitBake (activated by the -n or --dry-run command
line option), but it dry-runs the setscene tasks as well which we don't
want here.
Note that this has no effect on the checking we are doing with
BB_SETSCENE_ENFORCE to ensure that only whitelisted real tasks are
scheduled to run - that's handled separately.
Fixes [YOCTO #10369].
(Bitbake rev: 58f084291beb3a87d8d9fdb36dfe7eff911fa36b)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is more pythonic and can handle unclosed file warnings better
than the previous code structure.
(Bitbake rev: 50633012a64a3b5f0662145e29ff426374fb7683)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some of our metadata assumes that BuildStarted and BuildCompleted events
see the same data store. This is the case for buildTarget but not for
buildFile and recent changes mean this is now a problem.
The update_data() call is now an empty operation and there is no difference
between the expanded_data and data so we can simply remove the expanded_data
and its references and use data everywhere. This has been inteded for a while
but the above issue makes this more pressing to finally clean up.
(Bitbake rev: e3694e738e98f26f413ada6860ca7d829d3662f0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.
Search made with the following regex: getVar ?\(( ?[^,()]*), True\)
(a follow on patch to fix up a few recent introductions)
(Bitbake rev: f9961fd5beb31d5ab9656a5be59f7ab3effef2f0)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In certain circumstances it can be useful to get access to BitBake's
logger within a task; the main example is in OpenEmbedded's image
construction code where we want to be able to check the log file for
errors and warnings, but we don't want to see any errors or warnings
that were emitted through the logger; so we need a way to exclude those.
In order to do this, pass the logger object into the task via a
BB_TASK_LOGGER variable, and add a logging handler class to bb.utils
that can be added to it in order to keep a list of warnings/errors that
have been emitted.
(Bitbake rev: f1cd6fab604f14d8686b1d783cbfe012d923ee42)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
On terminating the connection to the server, we were disabling SIGINT -
and this is executed on the UI side. I'm not sure whether the intention
here was to undo the SIGINT disabling we did in the server, and it was
just a mistake that it disabled rather than restored and it's run on the
wrong side, or whether we wanted to stop the user from breaking out of
the shutdown code - the commit message provides no clues either way.
Regardless, we do not want to permanently disable Ctrl+C here - it's
legitimate to terminate the connection to the server and then
re-establish it within the same process; at least currently, devtool
modify by virtue of using tinfoil in two separate parts of the code does
this, and the result of this disabling is that during the second tinfoil
usage we can potentially be parsing all recipes without the ability to
easily interrupt the process.
(Bitbake rev: 58c60a951229dcbd8253863fb24228d046c23f6e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you run the setVariable command to set variables then you end up
causing the basehash to not match the previously computed values, which
triggers error messages. These mismatches are expected, so add a means
of disabling them.
(Bitbake rev: 5a80c0e210f26526afbe8f266b7b1a9c03334967)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow the client to set variables with the setVariable command and have
those changes take effect when running tasks. This is accomplished by
collecting changes made by setVariable separately and pass these to the
worker so it can be applied on top of the datastore it creates.
(Bitbake rev: 69a3cd790da35c3898a8f50c284ad1a4677682a4)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The COW object used within VariableHistory can't be serialised itself,
but we can convert it to a dict when serializing and then back when
deserialising. This finally allows DataSmart objects to be serialized.
NOTE: "serialisation" here means pickling, not over XMLRPC or any other
transport.
(Bitbake rev: bbbb2a53d5decf3b613a92c4ff77c84bfc5d4903)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we want to use this function/command internally, we don't want this
warning shown.
(Bitbake rev: 5cfbb60833e7b12d698c1c2970c17ccf2a4971bf)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you're expanding a value that refers to the value of a variable in
python code, we need to ensure that the datastore that gets used to get
the value of that variable is the client-side datastore and not just the
part of it that's on the server side. For example, suppose you are in
client code doing the following:
d.setVar('HELLO', 'there')
result = d.expand('${@d.getVar("HELLO", True)}')
result should be "there" but if the client part wasn't taken into
account, it would be whatever value HELLO had in the server portion of
the datastore (if any).
(Bitbake rev: cbc22a0a9aadc8606b927dbac0f1407ec2736b35)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It's not really practical for us to parse recipes on the client side, we
need to do it on the server because that's where we have the full python
environment (including any "pure" python functions defined in classes).
Thus, add some functions to tinfoil do this including a few shortcut
functions.
(Bitbake rev: 8f635815d191c9d848a92d51fdbf5e9fd3da1727)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For the purposes of server-side parsing and expansion allowing for
client-side use of the datastore, we need a means of sending a datastore
from the client back to the server, where the datastore probably
consists of a remote (server-side) original plus some client-side
modifications. To do this we need to take care of a couple of things:
1) xmlrpc can't handle nested dicts, so if you enable memres and simply
try passing a serialised datastore then things break. Instead of
serialising the entire datastore, just take the naive option of
transferring the internal dict alone (as a list of tuples) for now.
2) Change the TinfoilDataStoreConnector object into simply the handle
(number) when transmitting; it gets substituted with the real
datastore when the server receives it.
(Bitbake rev: 784d2f1a024efe632fc9049ce5b78692d419d938)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rewrite tinfoil as a wrapper around the UI, instead of the earlier
approach of starting up just enough of cooker to do what we want. This
has several advantages:
* It now works when bitbake is memory-resident instead of failing with
"ERROR: Only one copy of bitbake should be run against a build
directory".
* We can now connect an actual UI, thus you get things like the recipe
parsing / cache loading progress bar and parse error handling for free
* We can now handle events generated by the server if we wish to do so
* We can potentially extend this to do more stuff, e.g. actually running
build operations - this needs to be made more practical before we can
use it though (since you effectively have to become the UI yourself
for this at the moment.)
The downside is that tinfoil no longer has direct access to cooker, the
global datastore, or the cache. To mitigate this I have extended
data_smart to provide remote access capability for the datastore, and
created "fake" cooker and cooker.recipecache / cooker.collection adapter
objects in order to avoid breaking too many tinfoil-using scripts that
might be out there (we've never officially documented tinfoil or
BitBake's internal code, but we can still make accommodations where
practical). I've at least gone far enough to support all of the
utilities that use tinfoil in OE-Core with some changes, but I know
there are scripts such as Chris Larson's "bb" out there that do make
other calls into BitBake code that I'm not currently providing access to
through the adapters.
Part of the fix for [YOCTO #5470].
(Bitbake rev: 3bbf8d611c859f74d563778115677a04f5c4ab43)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In memory resident mode we don't really want to actually shut down since
it's only the client going away.
(Bitbake rev: 74db369c46043116359101cab70486afd82372c0)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows you to maintain a local reference to a remote datastore. The
actual implementation of the remote connection is delegated to a
connector object that the caller must define and supply. There is
support for getting variable values and expanding python references
(i.e. ${@...} remotely, however setting variables remotely is not
supported - any variable setting is done locally as if the datastore
were a copy (which it kind of is).
Loosely based on an earlier prototype implementation by Qing He.
(Bitbake rev: a3edc3eefa2d03c4ad5d12187b32fa4dc495082a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you send back a string here you get "TypeError: 'str' does not
support the buffer interface" errors in bitbake-cookerdaemon.log and
"IncompleteRead(0 bytes read, 22 more expected)" errors on the client
side.
(Bitbake rev: 0d659a7dfe5fb096f8aa4380320f9e2a464b3cb5)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we're in observe-only mode then we cannot run commands that would
affect the server's state, including getSetVariable, so prevent that
from being called in observe-only mode.
(Bitbake rev: 2c5a8661430edebff67ab4a108995033d182b5d6)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow you to specify -q / --quiet more than once to reduce the messages
even further. It will now operate as follows:
Level Option Result
----- ------ ----------------------------------------
0 Print usual output
1 -q Only show progress and warnings or above
2 -qq Only show warnings or above
3+ -qqq Only show errors
(Bitbake rev: 6cf2582e17c28ca04f5cfb59858c4a9778c700d4)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is no "datasmart" member, only dataroot. This dates back to the
original implementation of variable history support - it's surprising we
haven't noticed the issue until now, but I guess it's rare to change a
copy of a datastore in a manner which using the old reference would
cause an issue.
(Bitbake rev: febd5534b07edfdef15cedb0578730c582c7373f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Remove the template context to JSON decorator function as this is
deprecated by having a proper REST API.
(Bitbake rev: b65a8193368ffa1d15af24a6acde8dce6bd4d383)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update js tests so that we use the new Layer REST API.
(Bitbake rev: 81764ce3ebf0d2fcb3dc6965f6f07931d39e5524)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Switch the front end to use the proper REST API for retrieving layer
information.
(Bitbake rev: 5ea25c49091f4d4b5007af948e063ed25ba5766f)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add the url to the Layer typeahead so that this can be used later on by
the front end code to look up layer details.
(Bitbake rev: d195f24a1b30ae8698bff5e87308347b9596a2e2)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a get API for returning information about layers.
(Bitbake rev: 3fa5170c98c699d2a7a8380b696fc599efcc5dee)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add the front end mechanism to load the typeahead for the git revision
field on importing a layer.
Also fix one indentation issue and update the js test.
(Bitbake rev: 28114be42174095b812a93d4b5a0e01e953d74f8)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we're importing a layer it's useful to suggest available git revisions of the
layers. This uses git ls-remote to fetch the revisions and then filter on this.
Caching is added for this typeahead to avoid having to fetch this
information multiple times in a single session.
[YOCTO #8429]
(Bitbake rev: a94ae3ad0cb07a52004143c6f86f371b9e330e9f)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix flake8 issues and remove redundant __init__ function definitions
from typeaheads (likely a copy and paste error).
(Bitbake rev: be1f9f48da480d813e3364815cb3e002ba70dd22)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Switch the importlayer.js to use the new REST API for importing a new layer.
(Bitbake rev: 6475fd7e0d2488bf300b75322f2c00297cd1440b)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add layer adding REST api and remove old views method.
(Bitbake rev: 0c8e41d2217fd568a84e857d1be230fcfd4bb5c7)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow passing none as the project id, this is convenient for layers
which belong to projects already and therefore have their own project
field.
Add documentation string to function
(Bitbake rev: 78dc7b9f1801e7f4c266ba1369e5706f177ddaa1)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a visual indicator when the data is changing in the table, for instance if
it's being re-ordered, searched or paginated.
[YOCTO #10104]
(Bitbake rev: 554c4992b33b77526b4b37c7484f1dd00032ddef)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes [YOCTO #10444]
Added a new section about syntax called "Line Joining". The
section describes how you can use the backslash character (\)
to joing stuff.
(Bitbake rev: a2768ecae7846d72a1bdb7cbbc5e8d242af854f6)
Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that we use python3, we should refer to pip3, not pip.
[YOCTO #10774]
(Bitbake rev: 99136f5f591deef0c96d9aea2dbea1c216f38121)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie
Aníbal Limón
Mark Asselstine
Matthew McClintock
Paul Eggleton
Ross Burton
Joshua Lock
Michael Wood
Scott Rifenbark