We have been seeing UnicodeDecodeErrors when handling the
ImagePkgList MetadataEvent in ORMWrapper's
save_target_file_information() if the event includes filenames
that include non-ASCII characters.
In the short term work around this by converting paths to the
unicode type when passing them to Django's ORM. This is a bit of
a hack but it's too late in the cycle to do anything more invasive.
[YOCTO #9142]
(Bitbake rev: f50fff03b3de02e73a3cc2eb9935f7c345dbddc4)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
buildinfohelper stores current Build object in its internal
state. Any changes to Build object will be lost if internal
state is not updated as current buildinfohelper code
saves Build object from internal state when build is
completed.
This bug causes incorrect build state when build is cancelled.
Updating internal state should fix it.
Note, that this commit updates internal state after status of
the build is changed to Build.CANCELLED. There are several other
places in the code where Build object is updated without updating
internal state. They should be carefully analyzed and fixed.
(Bitbake rev: d056cf40fc55530cb1736aedfb9a3c355884991e)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When bitbake doesn't need to build anything it still sends
ImagePkgList event with empty 'pkgdata', 'imgdata' and 'filedata'
fields. This causes crash in buildinfohelper code as it's assumed
that above mentioned fields always have data keyed by build target:
ERROR: u'core-image-minimal'
Traceback (most recent call last):
File "toasterui.py", line 423, in main
buildinfohelper.store_target_package_data(event)
File "buildinfohelper.py", line 1218, in store_target_package_data
imgdata = BuildInfoHelper._get_data_from_event(event)['imgdata'][target.target]
KeyError: u'core-image-minimal'
Fixed this by using dict.get method with empty dictionary as default
return value instead of trying to get value without checking if target
key is in the data.
(Bitbake rev: c39cc463e6d9594bf2c5ac8bb74e834f6f2cf7c8)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When a build is cancelled the build (action) is complete if it has been
caused the request being cancelled then update the build outcome
accordingly.
(Bitbake rev: d94d12914d351bf560b06d6f4e45c294b04ecaa3)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
toasterui exits event loop on one of the following events:
CommandCompleted, CommandFailed or CommandExit.
Unfortunately none of them come from bitbake when build fails.
This is normai if toasterui runs in observer mode. However, if it's
in build mode this causes toasterui to stuck in the infinite loop
waiting for new events.
The only event we can rely on is BuildCompleted as it always
comes from bitbake unlike 3 above mentioned events.
Modified the code to always shutdown toasterui in build mode
on BuildCompleted event.
(Bitbake rev: 9cd60f98b13cf7b1c518851a51e1cbaa596d8f81)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The keys 'started', 'ended', 'cpu_time_user', 'disk_io_read' and
'disk_io_write' were added to the event recently, so they don't
exist in the events generated by bitbake server from older releases.
Checking if task_to_update structure has these keys before using
them should fix build of older releases.
(Bitbake rev: 79611d0ea742263074fbb0bf5f1e39df75fd9f55)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
buildinfohelper.brbe is lost when buildinfohelper is closed.
This causes incorrect report of brbe when build is done.
Saved brbe attribute before closing buildinfohelper and used
it to report correct brbe.
Got rid of useless and confusing 'ToasterUI build done 1'
log message.
(Bitbake rev: 5d7cce0d0ed70f6b3ebd6cbad300d86964a13398)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
DepTreeGenerated event doesn't contain 'providermap' data in jethro.
Modified buildinfohelper to handle events without this data. This
should make it possible to handle jethro events coming from jethro
bitbake server by the latest buildinfohelper.
(Bitbake rev: f6dcb1c9967f042beae024146781cb8235a9e1f2)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Return value of self.BBServer.registerEventHandler differs between
jethro and master. To be able to build jethro toaster should be
able to communicate with jethro bitbake server i.e. it must work
with both old and new registerEventHandler call.
(Bitbake rev: f356c154016c428a3b53af61a075de6f14d9d1d9)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In current toaster code BRBE(build request:build environment) value
is passed from toaster to buildinfohelper through the 'SetBRBE' event.
Passing it through environment variable is easier as it doesn't
involve rpc communication between toaster and bitbake server.
It also eliminates the need in running bitbake observer process.
Added parameter 'brbe' to BuildInfoHelper.__init__
Used environment variable TOASTER_BRBE to set brbe for
buildinfohelper object.
(Bitbake rev: a0c8e2b309055e5927a8ff729d292ccaa69d0575)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It was used for workaround git 1.7.9.2 which was released in 2012 which
should not be existed on nowadays host, so remove it to avoid
confusions.
(Bitbake rev: 6140d0cc9aecf1029ca16fed47071dfcc92f4269)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Python 2's sqlite3 module defaults to returning Unicode strings for SQL
text queries, which could trickle down to other parts of bitbake code and
cause unexpected Unicode conversions. Using byte strings avoids this issue.
For example, the git fetcher's AUTOREV support caches HEAD SHA1's using
bb.persist_data, so sometimes the git command strings passed to fetch2's
runfetchcmd() were unicode, potentially causing UnicodeDecodeErrors when
it appended the values of environment variables containing non-ASCII chars.
[YOCTO #9382]
(Bitbake rev: 09623a0811c613a47a01ae465b822d8156faca30)
Signed-off-by: Daniel Klauer <daniel.klauer@gin.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
sstate.bbclass for example writes siginfo files to a separate location
but we need to read taint data from the standard path.
(Bitbake rev: da444c9761ee15a59ea8880e3f812a5d3f1a1aaa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The taint values need to be passed from the server to the workers to
ensure they see the same stamp values. Also ensure that the "nostamp:"
prefix isn't included in the checksum value to match the server
calculation. This ensures the checksums are all consistent.
(Bitbake rev: f80ba20e90f3746f7faee3e0ff7f249025fec8ee)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In theory all the information to recalcuate the task signatures was written
into the siginfo/sigdata files. In reality, some of the information was
written into the filename.
Firstly this patch duplicates that info into the file itself just for easy
of use since its small.
Secondly, we abstract out the existing "calculate the checksum" code for
the taskhash, and add a function to calculate the bashhash based on the
informaiton within the file.
Finally, we call these functions when we're writing out the data to check
that the data we're writing is consistent. I've found a couple of places
it wasn't and its good to know about these in advance, rather than having
a siginfo/sigdata file which a given hash in its filename but a contents
which give a different result.
This should all combine to avoid a certain class of checksum bugs making
it into world, and identifying problems in advance.
(Bitbake rev: 0f50a18d7a0ea0d68edd8e5217e29111f4b1ea0b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When I enabled debugging of the checksum code, I found the value calculated
from siginfo/sigdata files for do_fetch tasks never matched. This was due
to an error in the way the data was being stored for these, it wasn't ordered
correctly. This patch fixes things so the checksums calculated from
siginfo/sigdata files is correct when file checksums are present.
(Bitbake rev: 046c1be7594fae2eec3d1f242ba3e9a85f1a1880)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The real method is a few lines later, this one is incorrect and
just causing confusion. Remove it.
(Bitbake rev: a896f263300f069400eae533be0daf5dedf41c95)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The tests were skipped when running without network even though they
didn't require network. This commit also adds a test case for URLs with
ports in them (the ports should not be considered when doing trusted
network checks).
(Bitbake rev: 77747de6b20538063eef3b188489a35bef225359)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake would fail to classify the following URL as belonging to a
allowed network, because of the port number in the url.
BB_ALLOWED_NETWORKS = "*.example.com"
SRC_URI = "http://git.example.com:8080/foo.tar.gz"
Since protocols aren't specified in the BB_ALLOWED_NETWORKS variable,
it's reasonable to believe that this should work regardless of protocol
being used.
(Bitbake rev: ff603df23037e10fb2cfdf150429cba3f65072cd)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add additional metadata to the layer created for build history to be
able to identify the layer and recipe later on. Specifically this is the
branch and release to which the recipe and layer are associated with
enabling differentiation of two recipes which are local release and
master and 'master' release.
[YOCTO #8528]
[YOCTO #8545]
(Bitbake rev: 3deebd887bddbbd02fd9829a180aab494b1af7c4)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It's very possible that we added layer as:
BBLAYERS += "/path/to/meta/"
then there would be warning:
WARNING: No bb files matched BBFILE_PATTERN_core '^/path/to/meta//'
This patch can fix the problem.
(Bitbake rev: 2b1cb21d18fb18399e682021b866babeced9a4aa)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the code as it stands today it not possible to execute a python function
and get "normal" python exception handling behaviour. If a python function
raises an exception, it forces a traceback to be printed and the exception
becomes a FuncFailed exception.
This adds in a parameter 'pythonexception' which allows standard python
exceptions to be passed unchanged with no traceback. Ultimately we may want
to change to this convention in various places but at least it means we can
start to add sane functions now.
(Bitbake rev: 85cf22fd0ed26bb7dc7738ef2a10441891f38ae2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are some codepaths where the file checksum is verified and can
be found to mismatch but the 'rename' logic doesn't kick in. If code
relies on the presence of a file for the checksum having been checked
(e.g. uninative.bbclass) then it can be used when the checksum hasn't
matched.
Therefore rename the file whenever an invalid checksum is encountered.
(Bitbake rev: 69ef6c8a9db02bfa0e3fac72481ec26586a29a01)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prevent a hang when shutdown() is called during parsing (e.g. after
SIGINT). We must not append 'None' to the jobs queue. Otherwise the
worker loop inside Parser.realrun() may break out at the wrong point,
causing the results queue thread blocking bitbake indefinitely.
[YOCTO #9319]
(Bitbake rev: 7ebea3e9a60232222efa8a546a0ff28a53029949)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently bbappend files in a layer are applied in the order they're
found on disk (as reported by glob) which means things are not
deterministic.
By sorting the glob results, the order becomes deterministic, the parsing
order for .bb files also should be deterministic as a result of this change.
[YOCTO #9138]
(Bitbake rev: 3f8febc4212fbd3485ac9bdd4ac71b8fb0a05693)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before this patch, directory symlinks mathcing filename pattern (either
a file name or a glob pattern) were followed. However, directory
symlinks deeper in the search chain were omitted by os.walk(). Now
directory traversal behaves consistently, ignoring syminks on all
levels.
One reason for choosing not to "walk into" directory symlinks is that
dir symlinks in externalsrc.bbclass in oe-core are causing problems in
source tree checksumming.
[YOCTO #8853]
(Bitbake rev: 66dff37ebcd1dd14ebd6933d727df9cf0a641866)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If local.conf contains an invalid line, e.g.:
APPEND += " igor"
(note the leading space) then nasty tracebacks are shown which confuse the
user. Change so the parse error is simply shown without a traceback, improving
the user experience.
[YOCTO #9332]
(Bitbake rev: 148aa1fb45dcb37a756a08301a7daf270e753180)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When prefix is part of the version directory it need to ensure that
only version directory is used so remove previous directories if exists.
Example: pfx = '/dir1/dir2/v' and version = '2.5' the expected result
is 'v2.5' instead of '/dir1/dir2/v2.5'.
[YOCTO #8778]
(Bitbake rev: c760531c6dbf88135ab9f8e6f0784ccbf2cce1e4)
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Little improvement for reference tokens by names instead of index.
(Bitbake rev: e8ea15eeb1857ed4bb6337836bd2fb1f5dbb1bdf)
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We recently dropped lockfiles for file:// urls which in itself makes
sense.
If a file url redirects to something like an http:// mirror, we'd have
no lock taken for the original file and could race against others
trying to download the file. We therefore need to ensure there is a
lock taken in the mirror handling code.
This adds code to take such a lock, assuming it isn't the same lock
as the parent url.
(Bitbake rev: 913b6ce22cd50eac96e8937c5ffc704bfce2c023)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fixes a problem when the repository contains multiple levels of submodules via a resursive submodule init.
(Bitbake rev: dbafbe229360ffe5908b106a9c10e274712b9b17)
Signed-off-by: Derek Straka <derek@asterius.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently xmlrpc server implicitly sets itself into single use mode
when bitbake server is started with anonymous port (0) or no port is
provided in command line. In this mode bitbake shuts down xmlrpc server
after build is done. This assumption is incorrect in some cases.
For example Toaster uses bitbake in this mode and expects xmlrpc server
to stay in memory.
Till recent changes single use mode was always unset due to the bug.
When the bug was fixed it broke toaster builds as Toaster couldn't
communicate with bitbake server in single use mode.
Reimplemented logic of setting single use mode. The mode is explicity
set when --server-only command line parameter is not provided to bitbake.
It doesn't depend on the port number anymore.
[YOCTO #9275]
[YOCTO #9240]
[YOCTO #9252]
(Bitbake rev: afc0dd5c532684f6201b1e12bbf4c226ea19062d)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bb.event.ParseStarted event is not processed by toasterui, but
present in event list. This causes the following error:
WARNING: Unknown event: <bb.event.ParseStarted object at ...
and non-zero return code:
WARNING: Return value is 1
(Bitbake rev: 1cc102f3d83d9467a3a3c422254333796ba95605)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Remove the very verbose log dump from toasterui. This generates several
megabytes of not that useful debug information and actually hinders
finding the original exception.
(Bitbake rev: a21dc134bdce2c9eb5e47c770094660f0c45c398)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This makes it possible to prevent a recipe to be cached, and thus,
parsed every time.
Use with care.
[YOCTO #8853]
(Bitbake rev: 78335c1fbe5266116700c2413aac28b00423a75b)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use a constant to define the name for the toaster custom images layer;
this constant is then used to identify this layer in various places.
(Bitbake rev: 2540969ec71612af7f9041cadcc401513e9b357b)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update the upstream url used for testing cups versions after upstream website
changes.
(Bitbake rev: 5f06041d4936fc22297945bbbad7020bfa9083c6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes [YOCTO #9231]
npm when given an invalid registry URL with --registry actually goes and
fetches from the default registry, but this commit makes sure it goes to the
specified one.
(Bitbake rev: 7c849be7c70a5db4f66fe3041486abb923b5e4ee)
Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of trying one time with a timeout of 20 seconds try 4 times with
a timeout of 5 seconds, to account for a slow server start.
(Bitbake rev: 4a7fe63126dd8177baa5ad21e59e0bebeea8c596)
Signed-off-by: Lucas Dutra Nunes <ldnunes@ossystems.com.br>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The data available from buildstats is now more fine grained than
previously, so take advantage of that to enrich the data we save
against tasks:
* Store the CPU usage for user and system separately, and display
them separately.
* Disk IO is now measured in bytes, not ms. Also store the
read/write bytes separately.
* Store started and ended times, as well as elapsed_time. This
will enable future features such as showing which tasks were
running at a particular point in the build.
There was also a problem with how we were looking up the Task
object, which meant that the buildstats were being added to
new tasks which weren't correctly associated with the build. Fix
how we look up the Task (only looking for tasks which match the
build, and the task and recipe names in the build stats data) so
the build stats are associated with the correct task.
[YOCTO #8842]
(Bitbake rev: efa6f915566b979bdbad233ae195b413cef1b8da)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For some reason, the values for SRC_URI[md5sum] and SRC_URI[sha256sum]
were not being expanded. That lead to the following code not working
as expected:
SRC_URI = "http://.../${PN}-${PV}.tar.gz"
MD5SUM = "123abc..."
SHA256SUM = "abcd1234..."
SRC_URI[md5sum] = "${MD5SUM}"
SRC_URI[sha256sum] = "${SHA256SUM}"
(Bitbake rev: ba011470df0ea8bd89f01c0b02ec4b3969e60ce7)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
XMLRPCServer.single_use attribute was always set to False.
This caused xmlrpc server to keep running after build is done as
BitBakeServerCommands.removeClient only shuts down server if its
single_use attribute is set to True.
(Bitbake rev: 0a60b0928a0a746a60d2c2f294ff1903963c7086)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this you get a rather odd traceback instead of the proper
exception message.
(Bitbake rev: 2fe1826d3077eeda6cde433d3a1e6620f74e08dd)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The output of "npm view dependencies" isn't entirely JSON if there are
multiple results, but the code here was just discarding the output if
the entire thing didn't parse as JSON. Split the output into lines and
iterate over it, parsing JSON fragments as we find them; this way we end
up with the last package's dependencies since it'll be last in the
output.
Digging further, it seems that the dependencies field reported by "npm
view" also includes optional dependencies. That wouldn't be a problem
except some of these optional dependencies may be OS-specific; for
example the "chokidar" module has "fsevents" in its optional
dependencies, but fsevents only works on MacOS X (and is only needed
there). If we erroneously pull in fsevents, not only is it unnecessary
but it causes "npm shrinkwrap" to throw a tantrum. In the absence of a
better approach, look at the os field and discard the module (along with
any of its dependencies) if it isn't for Linux.
As part of this, we can reduce the calls to npm view to one per package
since we get the entire json output rather than querying twice for two
separate fields. Overall the time taken has probably increased since we
are being more thorough about dependencies, but it's not quite as bad as
it could have been.
(Bitbake rev: 436d67fe7af89ecfbd11749a6ae1bc20e81f2cc8)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
"2 || 3" is a valid version specification for a dependency in an npm
package.json file, but of course that looks like something else when
sent to a shell. Quote the version value to avoid this.
(Bitbake rev: bea0246831a46d943d2e27d6b38f6e498bd3413c)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Continue after processing BuildStarted event to fix
WARNING: Unknown event: <bb.event.BuildStarted object at 0x2554150>
(Bitbake rev: 12f1fb8c9b70fea0c9145f881bcceb8af32df6af)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Toasterui exits only if bitbake observer shuts down.
In build mode it should exit when build is done.
Made toasterui exit on bb.command.CommandCompleted,
bb.command.CommandFailed and bb.command.CommandExit events
when it's running in build mode.
(Bitbake rev: b11f9d6d3c2eb615335901e1dcea699daf3afb4c)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently toasterui works only in observer mode. This is
artificial limitation which was made to support current toaster
design. As we decided to stop using bitbake server we'll
need to run toasterui also in build mode.
[YOCTO #7880]
(Bitbake rev: d4b5796899c3ca5c7becd7322291afd8afb35a31)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently toasterui ignores return value of setEventMask
command, which created confusing difference between set of
events set by this command and the real set used in the code.
Checked if setEventMask succeeded. Print error message and
exit if it's not.
(Bitbake rev: 6e3f13ffb47102b5df2da91fbc3f5da3179245b2)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Executing setEventMask command when bitbake server is in readonly
mode causes runCommand to fail with the following error:
'Not able to execute not readonly commands in readonly mode'
Set readonly attribute for setEventMask command to make it working
for Toaster UI. This should not do any harm as this command doesn't
influence cooker state.
(Bitbake rev: 8a47d30b2555255fbf6049c5ed69b29664c32b17)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Removed events not used in the code from the list.
Added events that are used in the code.
(Bitbake rev: 16b14ec16049cc2040a60ad5fc95f6e19dda91a6)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Reformatted and reordered list of events to make changes
easily and see them clearly in the diffs.
(Bitbake rev: 42a2d1115f2b23dc063a3172285ca3be73cf70bb)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tarballs that are fetched down via npm repositories seem to often have
unknown headers. This doesn't affect our ability to extract the contents
though so we don't really care to see those warnings.
(Bitbake rev: b38975103e52a0c25e9ad9032c8cca1c47cbdcc2)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
npm allows you to specify a dependency as a Github username+path, or
omit the version entirely. You can hit these if you don't use a
shrinkwrap file, with the result that the code later fails due to the
output of "npm view" being empty; so handle this lazily by just ignoring
this part of the dependency if it's not really a version.
(Bitbake rev: 7b7a65c44dbdd5ba9366d4e2093f76df8758d546)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
No code changes, just fix to use four spaces.
(Bitbake rev: 66a9ee7d54ca9c25209f72da079f260ccdcc872a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This ui does not work in master, nor has it been updated for several years.
[YOCTO #9178]
(Bitbake rev: 9fad1d13eed1f725971e6d12d3977cd31e07019a)
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've been gearing up the Toaster web UI to replace the Hob (GTK+ based) UI
for some time now; Hob has basically been on life support for the past few
releases. As of late last month in master, Toaster has the capability to
select the packages in an image, removing the last thing that Hob could do
that Toaster couldn't.
To recap, the reasons why Hob is being removed include:
- The code is tightly woven into BitBake, making it fragile. This means it
needs significant QA and maintenance on an ongoing basis.
- Some of the implementation is not ideal; we'll be able to remove some cruft
from BitBake and OE-Core at the same time.
- It's GTK+ 2 based, not the current GTK+ 3.
- Toaster is now a much more capable UI and is being actively maintained
The discussion about removing hob can be found at:
http://lists.openembedded.org/pipermail/openembedded-architecture/2016-February/000082.html
(Bitbake rev: be2cceea159c6ca9111eff3df87b98513eab6d72)
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
npm-shrinkwrap will sometimes resolve a git URL which instead of a http url, in
this case go and grab the dist.tarball via npm instead of using the resolved
URL.
(Bitbake rev: eb53b927ff59aa19cf28bc46beb9f9a185a59990)
Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous commit breaks absolute pathnames in file:// urls, this
fixes it.
(Bitbake rev: b8113a1800687a37a26ac28deafdbafd74cc138e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When in SRC_URI appears file://dir;subdir=foo unpacker copies 'dir' to ${WORKDIR}, not
${WORKDIR}/foo as it should be.
These changes are fixing following bugs as well:
Bug 6128 - Incorrect wildcard unpack behaviour in fetcher
Bug 6129 - Local directories unpack to a different location than local files
(Bitbake rev: e659a3b0c2771679057ee3e13cd42e6c62383ff2)
Signed-off-by: Alexander Shashkevich <alex@stunpix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For some reason the enablement piece of the patch went missing, add it.
(Bitbake rev: 0270b5a3873ed0aeca3a66198c87a6164fb644b8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
PN can contain '_', e.g. gcc-cross-x86_64 and an override cannot
hence we do this manually rather than use OVERRIDES.
(Bitbake rev: 7a6baf02617d1edced4eaff235e73d746e2a3b68)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For local files, there are no races with downloads, we don't need ".done"
stamps and we don't need lockfiles.
This considerably cleans up DL_DIR and all the pointless ".done" files
as well as removes stalls over local files with the same name.
(Bitbake rev: 48e903745db578d9b9b425a8d411c1369df0eb94)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than create ".lock" and ".done" files with no name, error,
forcing us to fix the cases where this is a problem.
(Bitbake rev: 81158071508cc68c39db7d501370872f44d335cc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we specify urls such as npm://somehost;someparams the fetcher currently
does a poor job of handling mirrors of these urls due to deficiencies in the
way decodeurl works. This is because "somehost" is returned as a path, not
a host.
This tweaks the code so that unless its a file url, the host is returned
correctly.
This patch also adds test cases for these urls to the exist set of test
urls.
We need to tweak the URI() class since this thinks this is a relative url
which is clearly isn't. We also need to handle the fact that encodeurl will
error if passed a url of this form (it would want the path to be '/'.
(Bitbake rev: 83203cd2e677706e0111892a7843b83263cb8bd9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we wget something which looks directory like we end up with lock files
and done stamps without names, they also all use the same lockfile.
This change ensures that we use separate lock files based on the url
and avoid creating the mysterious ${DL_DIR}/.done files.
(Bitbake rev: 20bc82086018832e047345a672d74b6c1c113650)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
npm fetcher with support for shrinkwrap files and lockdown files to easily
download and install an npm package with strict dependency resolution.
The SRC_URI should be in the format of:
SRC_URI = "npm://registry.npmjs.org/;name=${PN};version=${PV}"
To add a shrinkwrap and lockdown file use:
NPM_SHRINKWRAP := "${THISDIR}/${PN}/npm-shrinkwrap.json"
NPM_LOCKDOWN := "${THISDIR}/${PN}/lockdown.json"
(Bitbake rev: dec75bbc5d075acb322dad8b1c40d6bd518dc9fd)
Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is useful as npm-lockdown uses sha1 because npm releases the sha1 of
packages and whilst this is undocumented it seems no other algorithm is
supported
(Bitbake rev: fd5d9011f6dd7029895b64d8a02d33185b9aa8ae)
Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
At some point in the future, getVarFlag should expand by default. To
get there from the current position, we need a period of time where the
expand parameter is mandatory.
This patch starts that process. Clear errors will result from any code
which doesn't provide this. Layers can be fixed with an expression
like:
sed -e 's:\(\.getVarFlag([^,()]*, [^,()]*\)):\1, False):g' -i `grep -ril getVar *`
(Bitbake rev: aa3faebdf6af66ab34f74d328b2113de0b08c7ee)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
At some point in the future, getVar should expand by default. To get
there from the current position, we need a period of time where the
expand parameter is mandatory.
This patch starts that process. Clear errors will result from any code
which doesn't provide this. Layers can be fixed with an expression
like:
sed -e 's:\(\.getVar([^,()]*\)):\1, False):g' -i `grep -ril getVar *`
(Bitbake rev: fab717d303df0bcef737661f6917f275f35215a4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Define a new bitbake configuration variable BB_HASH_CHECKSUM_CACHE_FILE
that can be used to define the cache file to use for file checksum
cache.
(Bitbake rev: a965b390d6240e279c190b92b17c0573e9bd604c)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If no cache file name is given a default from class variable is used,
like before.
(Bitbake rev: 2602a312818f564961de7dfa63c429d45ff9e5ac)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move the local file checksum functionality from bb.fetch2 into
bb.checksum module.
(Bitbake rev: 4f60933283f377d68f191db849dac6c1dc7a0aed)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before this patch the usage of cache was quite useless as the file
checksums were not actually cached on disk but re-calculated every time.
This patch utilises the new writeout_file_checksum_cache() method of the
SignatureGenerator class to do the job.
(Bitbake rev: 5ac9cbf405841ed3f65e6f99a3cee032567fb182)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Extend the API in order to be able to write out the file checksum cache
onto disk. SignatureGeneratorBasic class now implements a method that
update the fetcher local files checksum cache with the task file
dependency checksums.
(Bitbake rev: ecdabd321d48fa367b89ebffc00aa525b6eaa95c)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Drop unused 'd' argument from the cache save methods, simplifying the
API.
(Bitbake rev: 81bc1f20662c39ee8db1da45b1e8c7eb64abacf3)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current behavior of Hob is that there is a "Run Image" button which becomes visible only for qemu images.
My suggested change is:
- if an image is selected and it is qemu-compatible, let the "Run image" button be named "Run qemu image"
- if an image is selected and it is not qemu-compatible, let the same button show up with the name "Run custom image", and besides that, an option shows-up to allow the selection of the custom script (by default it points out to runqemu script) to be used for launching this custom image
Note: in case there is more than one toggled image (qemu runnable or deployable), when the user clicks the "Run custom image" button, a dialog will be presented, allowing to choose between any of the existing images.
[YOCTO #8940]
(Bitbake rev: cc4cfc2370297b8feb2dc39d4262e73adf06c09a)
Signed-off-by: Mirela Rabulea <mirela.rabulea@nxp.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
An example prefix: `perl-5.22.1-r0 do_compile:`
(Bitbake rev: 792b759e59e31d2e43d525a6e50d866b4f51f072)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This lets us filter and use -l to show messages from that source specifically.
(Bitbake rev: 7946927156dec33364418988eb921ddb273660eb)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the worker segfaults, we may never see a TaskFailed event from it, only
a runQueueTaskFailed event. In this case, return_value isn't getting set
leading to an incorrect exit code from bitbake. Fix by setting return_value
in both places.
(Bitbake rev: e5dd50e0d95d532fe31dde61f8c6b1a7a72321e9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there is a missing provider and we're using "-k" mode alongside "-w",
we could get a traceback since there was no provider. Add tests to avoid this.
(Bitbake rev: 90a4805e4e770a433b4394ea99792731e9a4b546)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing issues where the self test, which uses tinfoil doesn't
notice the changed contents of include files. The issue is
cached_statements in the parser being reused when the files have changed.
Whilst looking at this, I realised there were some other issues:
* We need to also invalidate the mtime cache when cooker restarts
* We should pass full filenames to the file invalidation code
* We should process cached_statements as part of inotify invalidation
With these fixes, the caching is more reliable for memory resident
bitbake too. It does raise some questions about cache validation and
lifecycles and indicates bitbake does need more work in the area,
preferably with the removal of the globals. This at least highlights
and works around some of the current issues.
(Bitbake rev: 3f507ff8bc467fba936cf3f31bb8ea8e02f168e8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
CookerParser.shutdown code doesn't do all required work to shutdown
parser processes. As a result bitbake hangs if interrupted during
parsing. Putting None into the parser_quit queue should fix this issue
as it makes parsers to quit main loop.
(Bitbake rev: f67307977e8f089ce6d208d3e9de2a6a1768757e)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The brbe variable is set on the bitbake server when the Toaster
UI starts. This enables Toaster to associate events with the
correct build and build environment.
However, the brbe variable is also used when a build starts to
identify whether a new build needs to be created, or an existing
one looked up. This causes a bug for command-line builds which
happen after a Toaster-triggered build: because the brbe variable
is never unset on the server or the buildinfohelper, the new
command-line build events are treated as originating from the
previous build.
Ensure the brbe variable is reset when the buildinfohelper "closes"
a build, so that each build then either sets the brbe variable
(Toaster-triggered builds) or leaves it blank (command-line builds).
Also modify the localhostbecontroller so that the brbe variable
is not set on the server and not looked up from the server. This
ensures that it is only set when the triggerBuild() method is
called, and that it remains as None for command-line builds.
[YOCTO #9021]
(Bitbake rev: 4a6a8d0074f62208d843b06344be31ae73d9b745)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If something fails in a exec_func_python() the current stack trace shows
incorrect filenames and linenumbers. For example:
The stack trace of python calls that resulted in this exception/failure was:
File: '/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb', lineno: 200, function: <module>
0196: chksum = bb.utils.sha256_file(fn)
0197: f.write('%s\t%s\n' % (chksum, os.path.relpath(fn, baseoutpath)))
0198:
0199:
*** 0200:copy_buildsystem(d)
0201:
File: '/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb', lineno: 9, function: copy_buildsystem
0005:IMAGE_FEATURES += "splash package-management x11-base x11-sato ssh-server-dropbear hwcodecs"
0006:
0007:LICENSE = "MIT"
0008:
*** 0009:inherit core-image
0010:
0011:IMAGE_INSTALL += "packagegroup-core-x11-sato-games"
File: '/usr/lib/python2.7/subprocess.py', lineno: 535, function: check_call
0531: The arguments are the same as for the Popen constructor. Example:
0532:
0533: check_call(["ls", "-l"])
0534: """
*** 0535: retcode = call(*popenargs, **kwargs)
0536: if retcode:
0537: cmd = kwargs.get("args")
0538: if cmd is None:
0539: cmd = popenargs[0]
The problem is the use of "FILE" to obtain the current filename. Instead,
we therefore inject the function being executed into the methodpool which
allows us to correct its linenumber and filename information. We can then
clearly mark the initial piece as autogenerated and the rest of the linenumber
and filename information should be correct. Afterwards the trace starts:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:copy_buildsystem(d)
0003:
File: '/media/build1/poky/meta/classes/populate_sdk_ext.bbclass', lineno: 66, function: copy_buildsystem
0062: import glob
0063: import oe.copy_buildsystem
0064: import subprocess
0065:
*** 0066: subprocess.check_call("foo")
0067:
0068: oe_init_env_script = d.getVar('OE_INIT_ENV_SCRIPT', True)
0069:
0070: conf_bbpath = ''
File: '/usr/lib/python2.7/subprocess.py', lineno: 535, function: check_call
0531: The arguments are the same as for the Popen constructor. Example:
0532:
0533: check_call(["ls", "-l"])
0534: """
*** 0535: retcode = call(*popenargs, **kwargs)
0536: if retcode:
0537: cmd = kwargs.get("args")
0538: if cmd is None:
0539: cmd = popenargs[0]
We can't inject into methodpool at parsing time, since there may be
_append or other override operations against the function before its
execution.
(Bitbake rev: fae153095d23157dd7e72c29f683f86149ee33a8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Right now, if you have some python code like:
X = "a"
def somefunction(d):
d.setVar("X", "b")
d.setVar("Y", "${X}")
then any sane person would expect that Y = "b" at the end of the
function. This is not the case, Y = "a".
This is due to the python function being expanded before execution, the
executed code would read d.setVar("Y", "a"). This understandably
confuses people, it also makes it near impossible to write ${} in a
python function without unintended things happening.
I think there is general agreement we should fix this and standardise
on non-expansion of python functions. We already don't expand anonymous
python (mostly).
I've checked OE-Core with buildhistory before and after this change and
there were a small number of issues this exposed which I've sent
patches for.
I propose we default to not expanding python code and then deal with
any consequences from that if/as/where identified. This will improve
new user understanding and usability of the system, it also allows
several long standing weird expansion issues to be fixed.
(Bitbake rev: 8bf33a8e92c0e188fa392030025756196c96fcbb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We don't want to expand python functions since they aren't expanded
at execution time (e.g. anonymous python). They can also have side
effects.
This function is primarily used by toaster for variable dumps for later
display. The lack of expansion of python functions won't matter in this case
and actively helps some variable handling (e.g. SRCPV).
(Bitbake rev: 3f5520b4844a4bdd615046479ba08ed192bdc8cd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Expanding python functions for variable dependencies doesn't really make sense,
not least since this causes execution of any inline python, it also makes it
impossible to write expressions like d.expand("${X}") of d.setVar("X", "${Y}")
which may have the wrong values if expanded now.
This starts to standardise the approach across bitbake for handling python code.
(Bitbake rev: 765a2480dbe288f64562a9611dd93b6b6dd0a64e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We don't expand anonymous python before execution, so nor should
we do this when calculating checksums for them.
(Bitbake rev: 5f10987edda35b08970a6dd6ccf9febad271ce3e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The functionality overlap between these two functions is significant and
its clearer to handle both things together since they are intimately
linked. There should be no behaviour change, just clearer code.
(Bitbake rev: 391aa4afc91be90d8d3ee47e1bf797d6ebe61a71)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we use functions from the data store, they now have correct line number
and filename information. This function would attempt to correct line numbers
which doesn't need correcting, leading to misleading messages to the user.
Therefore remove this code as being obsoleted.
(Bitbake rev: 918bec86bc8ee94feb82380ff410d9fdcbe9e720)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Extended the dot styling of dependencies created by bitbake -g in dot syntax to differentiate between the various kinds.
depends: solid
rdepends: dashed
rrecommends: dotted
The change observed is that depends get an explicit style which is the same as dot default behavior and the runtime recommends get
dotted while before they were dashed. This helps to distinguish them graphically as well as eases post processing by script.
(Bitbake rev: 86e78e0ca7aa5452411f35239942ecee3d8824ec)
Signed-off-by: Henning Schroeder <henning.schroeder@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of keeping the original dependency information for the pool of
CustomImagePackage reset it with each new build.
(Bitbake rev: a0b97ffc7a468bad081ce3276c74728bf6830250)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds the concept of CustomImagePackage this is similar to the way
layers and recipes work in that we have a set of data which is part of
the build history and a set of data which is part of the configuration
data that toaster uses to guide people in configuring their project. We
create a set of built_packages for every build but only create a package
for configuration purposes if we don't already have one, so that the
CustomImagePackage only ever contains a unique list of packages that are
available to be added and removed from a CustomImageRecipe.
(Bitbake rev: f81bb65883baa6c0f8a4d48a4de3291a10543992)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to use in other modules since is a common function
when needs to get proxies working.
(Bitbake rev: 85c529044381895556d603a3974de22392646a22)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently any not catched exception in cooker causes bitbake
to hang because of not terminated children of CookerParser.
Long term solution would be to reimplement Cooker as a context
manager and terminate parser children in its __exit__ method.
Partial fix is to call CookerParser.shutdown in Cooker.shutdown in
hope that all Cooker exceptions are caught and shutdown method is
called.
[YOCTO #8900]
(Bitbake rev: 3f67600dc3292bc8208644ce89e8bf7ab95cf2e7)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since people do copy and paste these things, clean up old syntax styles.
(Bitbake rev: 4fb028b0bd14d3e4b3fd7a89c643528728566476)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This shouldn't be in here, use a variable instead.
(Bitbake rev: 2e25d09a1ab62ccc3573d13114d59838cf4b07f2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Anonymous functions are python functions, set the variable
flags as such so we can detect them and avoid expansion where
needed.
(Bitbake rev: 1b303785c578bbae3a89be8d751d80fba860f62e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently bitbake doesn't parse into data.expand() expressions,
relying on high level expansion of python code to handle this.
One of the tests does however test this works.
We don't really want to be doing string expansion on python code,
so specifically parse into expand() function calls so that when
the high level behaviour is tweaked, the self tests continue to
pass and that we do continue to handle expand() function calls as
best we can.
(Bitbake rev: b12c17be5e4a74c9680876605c87f46501f78d28)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This sets the scene for removing the default False for expansion from
getVarFlag. This would later allow True to become the expand default.
On the most part this is an automatic translation with:
sed -e 's:\(\.getVarFlag([^,()]*, [^,()]*\)):\1, False):g' -i `grep -ril getVar *`
There should be no functional change from this patch.
(Bitbake rev: 7c3b99c6a716095af3ffce0b15110e91fb49c913)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fix a problem when checking out a commit that changes the submodules
previously checkout.
Example:
Recipe uses branch A and then it updates to use branch B, but branch B has
different submodules dependencies then what branch A previously had.
(Bitbake rev: 54a3864246f2be0b62761f639a1d5c9407aded4f)
Signed-off-by: Felipe F. Tonello <eu@felipetonello.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I before E, except after C...
(Bitbake rev: 14c9593265f7469cb8a205a46f845ac7491246df)
Signed-off-by: Phil Blundell <pb@pbcl.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we're only going to parse one recipe, no point in starting
a large number of threads.
(Bitbake rev: b977faf59dc08050a44a16032fe52d1bbb80f2a1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When running universe builds, we don't expect an error exit code for
provider warnings. Change the error messages to warnings in this case.
This deals with errors causing problems on our autobuilders amongst
other issues.
(Bitbake rev: d4989fb0355476de172169f0698757f7360e9a1f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
findServerDetails function can be removed safely
from the source tree. Couldn't find any files
calling this function.
(Bitbake rev: 46871f769db13ccd36deedc5b6f3dbc0a3d31c4b)
Signed-off-by: Sujith H <sujith.h@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The recent change to verify_checksum() to only show checksum warnings
if no checksums are supplied made it possible to simplify the logic a
bit more.
(Bitbake rev: 1dc00b874acae44bbba9d8028d94f7bc97ddcd76)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This means that when you view the process tree, the processes
have meaningful names, aiding debugging:
$ pstree -p 30021
bash(30021)───KnottyUI(115579)───Cooker(115590)─┬─PRServ(115592)───{PRServ Handler}(115593)
├─Worker(115630)───bash:sleep(115631)───run.do_sleep.11(115633)───sleep(115634)
└─{ProcessEQueue}(115591)
$ pstree -p 30021
bash(30021)───KnottyUI(117319)───Cooker(117330)─┬─Cooker(117335)
├─PRServ(117332)───{PRServ Handler}(117333)
├─Parser-1:2(117336)
└─{ProcessEQueue}(117331)
Applies to parse threads, PR Server, cooker, the workers and execution
threads, working within the 16 character limit as best we can.
Needed to tweak the bitbake-worker magic values to tell the
workers apart.
(Bitbake rev: 539726a3b2202249a3f148d99e08909cb61902a5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Being able to tell the bitbake processes apart is useful for debugging.
Add a helper function which allows this without making it a hard
dependency. Errors are ignored, this is just nice to have.
(Bitbake rev: fd7f1a94d196b8a3c445e313d9e699b352b1da97)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A newline is always appended to the function body when it's written
out, so strip any trailing newlines which may be there already.
(Bitbake rev: 8a3f50936113e15d2f2822f6aee494204fa1c24f)
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix quoting of $BASH_COMMAND and avoid wrapping at 80 columns (the
script which follows is likely to contain some very long lines, so
line wrapping in bb_exit_handler() looks somewhat out of place).
(Bitbake rev: 8e12c8f8441a7c6a03e603c5789d6037945704c1)
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before, BBMASK was only permitted to contain one regular expression.
This made it hard to add to the BBMASK in multiple places as one was
supposed to separate the different regular expressions with a "|"
rather than with whitespace as is customary in BitBake variables.
Now one can specify any number of regular expressions in BBMASK. This
makes it possible to, e.g., mask out recipes in another layer from the
layer.conf file.
This also properly ignores any regular expressions that do not compile
(before an invalid regular expression would cause a ParseError in the
first bbappend file found stating that it was not a BitBake file...)
(Bitbake rev: 2c778ad50aceaffb855baf5f4aa0fed98c880870)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The core change here is to fall back to GET requests if HEAD is rejected in the
checkstatus() method, as you can't do a HEAD on Amazon S3 (used by Github
archives). This meant removing the monkey patch that the default method was GET
and adding a fixed redirect handler that doesn't reset to GET.
Also, change the way the opener is constructed from an if/elif cluster to a
conditionally constructed list.
(Bitbake rev: 6ec70d5d2e330b41b932b0a655b838a5f37df01e)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If one checksum is supplied to a SRC_URI, we really don't want to show
warnings about the other type which isn't present as one checksum
is really good enough for most cases.
(Bitbake rev: 43358a9b595b2928458a5f463cf1949394160c3a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In certain circumstances it can be useful to get access to the world
targets list from a recipe in order to add dependencies on some or all
of the items in it. If a special function, 'calculate_extra_depends' is
defined in the recipe, and the recipe is to be built, then call it at
the right point before we calculate which tasks should be run. The
function can append items to the "deps" list in order to add
dependencies. This is not as tidy a solution as I would have liked, but
it does at least do the job.
As part of this change, the buildWorldTargets function was moved to
bb.providers to make it possible to call from taskdata.
Part of the implementation of [YOCTO #8600].
(Bitbake rev: aba0dce57c889495ec5c13919991a060aeff65d2)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The rprovides maybe contain duplicated lines when parse again, we need
check it before add to cachedata.rproviders, similar to what we had done
to cachedata.providers.
(Bitbake rev: 6c488afb0fe30a9655ec62a1d22f9f388365f012)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is useful for newbie, for example:
$ bitbake rpm-build
ERROR: Nothing PROVIDES 'rpm-build'. Close matches:
pm-utils
rpm RPROVIDES rpm-build
[YOCTO #8881]
(Bitbake rev: 4b59eb8cc2321fe72f2988b6c9c0fecd4883255b)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If variables are unset, the code simply doesn't expand them, there
aren't errors. If the code is a python expression, this can get a bit
messy, see the attached test case. The python expansion code sees the }
of the unexpanded value rather than the close of the python expression
and then raises a SyntaxError exception.
Ideally, we'd update the code to match pairs of brackets. I don't know
how to do that with the current regex and this is unfortunately a
performance sensitive piece of code. We also run the risk of breaking
existing code in OE-Core where there are "{" characters but not "}"
to close them (PKGE and PE).
Rather than raising the exception, matching the existing "just return
the expression" behaviour seems more consistent with the standard
variable behaviour.
This addresses an issue found in the recent image.bbclass code where
there are some variables we choose not to expand (TMPDIR/DATETIME).
This patch also adds a test case for this behaviour. It wouldn't preclude
improved bracket matching code in the future either.
(Bitbake rev: d80d39e73223a50fda0090784303d2c57167bb4c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A recent change in bitbake added filename/lineno information to the
parameters of bb.data.build_dependencies(). The codeparser tests
required a little adaption to the changes, adding the flags to the FOO
variable used in the tests.
The error seen when running the tests is a TypeError exception raised
in bb.codeparser:
TypeError: int() argument must be a string or a number, not 'NoneType'
(Bitbake rev: f1fe674397ac5cd355696d5b4cc90b7cfa6c867f)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit b22592af81.
That commit isn't entirely clear about why this change is needed but
I do have a usecase where this breaks things. If for example you run
"bitbake X -c packagedata" and that packagedata is in sstate, you'd
expect this to work.
If sstate doesn't contain a do_populate_sysroot for a dependency, you
would still expect the command above to succeed and you would not
expect
it to rebuild that dependency. With the current code, this isn't what
happens. The code finds the sstate for do_populate_sysroot missing,
this makes the task "uncovered" and this in turn makes it unskippable.
The example I found with this was avahi-ui, where it would trigger
a build of libdaemon to obtain its populate_sysroot.
Since this behaviour seems completely incorrect, revert the older patch
and we'll address any issues that crop up as a result.
(Bitbake rev: 36a9840a5da17cc14561881fdd6a4f2cb0a75e49)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
OSErrors occurring in toaster.bbclass are converted to
OSErrorException metadata events. They were then being swallowed
as unprocessed events by toasterui, which made them difficult
to spot.
Explicitly catch OSErrorException events and log them so they
are easier to spot and debug.
(Bitbake rev: 69f2b2bc373ce114609600b59a6b6ccef20771c9)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The event mask for toasterui doesn't include MetadataEvents.
This means that we're missing the ArtifactFileSize event
(among others), which is the one we use to populate the SDK
artifact table.
Add that event type to the toasterui event mask so we can
record SDK artifacts as they are created.
[YOCTO #7603]
(Bitbake rev: d0276a831bb8cffd42c8367895633eaa1fa1ed30)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit 4193e99adce8e88f12ac88d7578ad39575f7e346.
It seems the underlying issue was caused by ":" in the url which isn't
supported. The patch was therefore incorrect.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit e130dca85bac82bd4d88f94a6bf9fe36e8ad4d7c.
This is in fact a valid use case, for example the sstate.bbclass code
sets up SSTATE_MIRRORS as PREMIRRORS. Its quite common to map those
file:// urls to remote http:// urls and with the above change, this
no longer works.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This goes undetected most of the time, but when updating a repository,
if the ud.fullmirror file is not present, you end up getting an
exception instead of carrying on because the errno module is not
loaded (specifically "if exc.errno != errno.ENOENT").
(Bitbake rev: e6fca8480731ce817df9bee61438347a5e3d3017)
Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The 'stamp-base' and 'stamp-base-clean' related codes are no longer useful,
clean them up.
[YOCTO #8468]
(Bitbake rev: 7b4c42b315d4a26dd8f2ceb874a94737bf9f183e)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Debugging suggests that setscene tasks are being a little greedy about their
dependencies, for example, lsof is insisting that gcc-runtime's do_package
is installed. If it isn't, its requiring gcc to rebuild.
If gcc-runtime do_package_write_xxx and do_packagedata is available, there
is no reason do_package should be needed.
The reason this is happening appears to be from the batching up of task
dependencies code, rather than setscene tasks stopping when passing over
a setscene task, they were being carried forward. This patch fixes it
so the data is 'zeroed' when passing over a setscene task boundary,
which gives the dependency graph that is expected.
After this patch, lsof will rebuild quite happily without
gcc-runtime:do_package being present, as expected. This should lead to
less dependencies being installed for builds from sstate and generally
better performance in general.
(Bitbake rev: f8bcb0a1e3b008b71c9a7cd21f76d0906f2d8068)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Added new entries to Provides model and link them to
Recipe_Dependency using 'via' field.
This data will be used by Toaster UI to show 'Provides:'
information for the recipes.
[YOCTO #6169]
(Bitbake rev: 336ddc8df611d4c8f1c3d3a06d0a85bb544c38bc)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Used providermap in store_dependency_information function
to find virtual dependencies. This should fix annoying
warnings "stpd: KeyError saving recipe dependency"
[YOCTO #6169]
(Bitbake rev: 85c416ca338c886db6e79651e44727482df9fb07)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Added providermap information to the result of buildDependTree API.
This will be used by Toaster to map virtual dependencies to recipes.
(Bitbake rev: d3e07368549f30265f59846a260efa8230a225ca)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Added optional parameter 'prefix' to filter out names that
don't start with specified prefix. Changed existing call
of get_providermap according to changed API.
Optimized the code: got rid of extra loop and temporary
list variable virts.
(Bitbake rev: df5a1392d6f91ccb44a99721c7d847da242121bb)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its turning out that we really need a way to have bitbake just run
the setscene tasks but not any real tasks, particularly for SDK
operations.
Add an option for this since its pretty straight forward. This allows
various nasty workarounds in OE-Core to be removed.
(Bitbake rev: e4a2aafa1650a227a04d92a8a0b31efaed2c310e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some servers, e.g. bitbucket.org can't cope with ssh:// as part of
the git url syntax. git itself is happy enough with this but you
get server side errors when using it.
This changes the git fetcher to use the more common ssh url format
which also means we need a : before the path.
Seems a shame to have to do this due to broken servers however
it should be safe enough since this other form is the one most people
use on the commandline so it should be safe enough.
[YOCTO #8864]
(Bitbake rev: 4193e99adce8e88f12ac88d7578ad39575f7e346)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
TERM=dumb bitbake X
shows no output for task status which is suboptimal. Use the non-interactive
mode if the terminal doesn't support what we need for interactive mode giving
a better user experience. Also print a note to the console to say this has
happened.
[YOCTO #8768]
(Bitbake rev: 6f84cf4bd77f35fcd07e0b2f5149f1d6866a414d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The documentation for findFilesMatchingInDir() was inconsistant with the
implementation: the regex was escaped before searching so effectively it's a
pure textual substring, and the machine example was broken.
(Bitbake rev: 6bef981488ec94b46dbe3797acfecf9c4b6ecbbc)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
No point counting all instances when we just want to know if there's any or not.
(Bitbake rev: 3369072efb653339da8dbd1ca864ff8e1ff899ca)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Where we add in mappings for EXPORT_FUNCTIONS, add dummy filename
and lineno data so ensure the assumption that all python functions
have this is correct.
(Bitbake rev: 547128731e62b36d2271c4390b3fee2b16c535dc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Setup of event queue includes registering of UI handler.
This operation can fail when cooker is busy. However, there is
no need in registering UI handler for terminating the server.
Moved the call of connection.terminateServer before setting up
of the event queue. This should make terminating server to work
more reliably as it doesn't depend on setting up the event queue
and registering UI handler anymore.
This should also help Toaster backend to restart bitbake server
and observer without getting "Could not register UI event handler"
errors.
[YOCTO #8776]
(Bitbake rev: 0c5a9349f797d05c282c2ada1893e187e05f0576)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Current code in connect method sets up event queue, which requires
registering UI handler. This functionality may not be needed for
some operations, e.g. for server termination.
Moved functionality of setting up event queue in from 'connect'
method to 'setupEventQueue' in BitBakeXMLRPCServerConnection class.
(Bitbake rev: 4429871da76d6bd29e023ff42740fe7daa6b40fa)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Replaced 'while' loop with 'for' loop.
Made the code more compact and hopefully more understandable.
(Bitbake rev: 4e1e497c8432536b3522295e5b1284844ccea056)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This attribute was introduced by mistake. EventHandle is used in the
code for the same purpose.
(Bitbake rev: 8d505ec8913a7d51de48b4f52bb64c5d6a0bb08e)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Current code throws Exception("Could not register UI event handler")
if event handler can't be registered. The real reason of this is that
cooker is in busy state. Error message lacks information about this.
Added error message to the return value of registerEventHandler.
Included returned error message into the log message and exception
text.
(Bitbake rev: 07de1ca7d57dcd0cc37406feae2949da12a3fa7a)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Added method to convert state code into the human readable name.
It will be used in logging and error reporting.
(Bitbake rev: 9ec6379b27d210214d0b3f2e55962f721b7f5f51)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
7-Zip is a file archiver claiming the highest compression ratio.
This patch allows using 7-Zip commpressed files in bitbake recipes.
Two common formats are supported:
SRC_URI = "file://abc.tar.7z"
SRC_URI = "file://abc.7z"
(Bitbake rev: 7120f5bfaae54e91bc95da5667831424724ce613)
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Poking around the ast to correct linenumbers works well for runtime failures
but not for parsing ones. We can use blank linefeeds to correct the line
numbers instead, with the advantage that we don't need to double compile.
(Bitbake rev: 10256ac3e7be7e691176ecc5d55856d88f1fe940)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The PREMIRROR isn't useful for "file://", so avoid using it, this is
good for searching speed and can reduce useless lines in log.do_fetch.
(Bitbake rev: e130dca85bac82bd4d88f94a6bf9fe36e8ad4d7c)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This makes no sense as just a note, its at least a warning and useful
to get an idea of which codepath is failing.
(Bitbake rev: 0194cf0da24dc72dab0612cd54aa5190e6cd92f2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This codepath can be triggered by a python indentation error for example.
Showing it as an ExpansionError is misleading.
Change the code to add a warning about where the failure came from (in
particular giving the variable key name that triggered it) but raise the
proper exception.
(Bitbake rev: d49d46533704e8b4404e29abfb5a7383d704c91a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The SyntaxError exception simply shows the recipe that failed to parse
which is pretty useless without the actual exception. We could make it
print more info, however we can just use one of the more generic handlers
instead and remove this one.
For a python indentation error, this leads to a much more readable error
message.
(Bitbake rev: 9241eb10847634e34c5ff8767ed8c114f66ff6cf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If parse_python() fails, the output is confusing. Passing in the extra
file/line data isn't expensive and improves readability significantly.
(Bitbake rev: a4bb753488d322e0e31c31d6377ba780f2f824c4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if there is something like a python indentation error in a
python function, the linenumbers and even file aren't reported correctly.
This allows lineno and filename parameters to be passed in to correct this.
The lack of a lineno parameter to python's compile() function is worked
around by using empty linefeeds. Ugly, but effective and with minimal
performance overhead.
(Bitbake rev: 5796ed550d127853808f38257f8dcc8c1cf59342)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the addition of function line number handling, the overhead of
the compile functions is no longer negligible. We tend to compile
the same pieces of code over and over again so wrapping a cache around
this is beneficial and removes the overhead of line numbered functions.
Life cycle of a cache using a global like this is in theory problematic
although in reality unlikely to be an issue. It can be dealt with
if/as/when we deal with the other global caches.
(Bitbake rev: 98d7002d1dca4b62042e1589fd5b9b3805d57f7a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now we're actively using the line numbers for other thins, having
magic values like IN_PYTHON_EOF causes problems, in particular, 32
bit overflow on 32 bit machines.
There is a neater way to signal eof to feeder(), just using an extra
parameter so use this instead and drop the IN_PYTHON_EOF magic values.
This has the added bonus that line numbers are then correct for
python functions at the end of files.
(Bitbake rev: e0f05871c2a6f1e86ae19ad343c7c6f822ddb67e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of:
"""
can only concatenate tuple (not "int") to tuple
"""
we now see:
"""
Traceback (most recent call last):
File "/media/build1/poky/bitbake/lib/bb/ui/knotty.py", line 324, in main
termfilter.updateFooter()
File "/media/build1/poky/bitbake/lib/bb/ui/knotty.py", line 210, in updateFooter
lines = 1 + int(len(content) / (self.columns + 1))
TypeError: can only concatenate tuple (not "int") to tuple
"""
which makes tacking down and fixing the problem much easier.
Also ensure we set an error exit code.
(Bitbake rev: d965bcae6cfd268406a3bd1ef77c5bb6c6e1c6d7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the row handling was introduced, one of the callbacks was
missed resulting in:
TypeError: can only concatenate tuple (not "int") to tuple
Fix it.
(Bitbake rev: 0b77cea2bf5b5f5704e2650fb0332f5d78037781)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This updates buildinfo helper for the recent buildstats layout change
(Bitbake rev: 30311bbe667e9f22de17fae00ff58da06a7c3e23)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When a mirror tarball is fetched, the original fetch method is called, which
unpacks the mirror tarball. After the original method is called, it checks the
localpath of the mirror tarball rather than the clone path, which isn't ideal,
particularly if the mirror tarball was removed due to being out of date. We
know the original fetch method will do what it needs to do to get its content
in the form it needs from the mirror tarball, so we can use its localpath
instead.
(Bitbake rev: 1732ad65d6c7d67b7d07cb30c074f5016adadbea)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Awais Belal <awais_belal@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you break the internals of better_exec(), you get a very weird
error about tb_next not being a method of None. Fix this by checking
we can step back a trace level.
(Bitbake rev: 1d710ed484f68fca0789022dde7ba877b9a894f5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently bitbake tracebacks can have places where the line numbers are
inaccurate and filenames may be missing. These changes start to try and
correct this.
The only way I could find to correct line numbers was to compile as a
python ast, tweak the line numbers then compile to bytecode. I'm open
to better ways of doing this if anyone knows of any.
This does mean passing a few more parameters into functions, and putting
more data into the data store about functions (i.e. their filenames
and line numbers) but the improvement in debugging is more than worthwhile).
Before:
----------------
ERROR: Execution of event handler 'run_buildstats' failed
Traceback (most recent call last):
File "run_buildstats(e)", line 43, in run_buildstats(e=<bb.build.TaskStarted object at 0x7f7b7c57a590>)
NameError: global name 'notexist' is not defined
ERROR: Build of do_patch failed
ERROR: Traceback (most recent call last):
File "/media/build1/poky/bitbake/lib/bb/build.py", line 560, in exec_task
return _exec_task(fn, task, d, quieterr)
File "/media/build1/poky/bitbake/lib/bb/build.py", line 497, in _exec_task
event.fire(TaskStarted(task, logfn, flags, localdata), localdata)
File "/media/build1/poky/bitbake/lib/bb/event.py", line 170, in fire
fire_class_handlers(event, d)
File "/media/build1/poky/bitbake/lib/bb/event.py", line 109, in fire_class_handlers
execute_handler(name, handler, event, d)
File "/media/build1/poky/bitbake/lib/bb/event.py", line 81, in execute_handler
ret = handler(event)
File "run_buildstats(e)", line 43, in run_buildstats
NameError: global name 'notexist' is not defined
----------------
After:
----------------
ERROR: Execution of event handler 'run_buildstats' failed
Traceback (most recent call last):
File "/media/build1/poky/meta/classes/buildstats.bbclass", line 143, in run_buildstats(e=<bb.build.TaskStarted object at 0x7efe89284e10>):
if isinstance(e, bb.build.TaskStarted):
> trigger = notexist
pn = d.getVar("PN", True)
NameError: global name 'notexist' is not defined
ERROR: Build of do_package failed
ERROR: Traceback (most recent call last):
File "/media/build1/poky/bitbake/lib/bb/build.py", line 560, in exec_task
return _exec_task(fn, task, d, quieterr)
File "/media/build1/poky/bitbake/lib/bb/build.py", line 497, in _exec_task
event.fire(TaskStarted(task, logfn, flags, localdata), localdata)
File "/media/build1/poky/bitbake/lib/bb/event.py", line 170, in fire
fire_class_handlers(event, d)
File "/media/build1/poky/bitbake/lib/bb/event.py", line 109, in fire_class_handlers
execute_handler(name, handler, event, d)
File "/media/build1/poky/bitbake/lib/bb/event.py", line 81, in execute_handler
ret = handler(event)
File "/media/build1/poky/meta/classes/buildstats.bbclass", line 143, in run_buildstats
trigger = notexist
NameError: global name 'notexist' is not defined
----------------
(Bitbake rev: 1ff860960919ff6f8097138bc68de85bcb5f88b0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It can be useful to run all tasks up to but not including a specific task. The
main reason this was never added was the lack of a good syntax. This patch
uses the syntax <taskname>- to denote this behaviour which is simple, not
invasive and fits what we need from good syntax IMO, hence we can add this.
(Bitbake rev: 99ccfd411ab3f7baa111f9f3d50fae68816a9a83)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
buildinfohelper, with the new import paths for our Django models
and Django 1.8, was not getting an active connection to the database.
In buildinfohelper, call django.setup() explicitly to make sure
that the database connection is ready and models can be queried
and saved.
[YOCTO #8364]
(Bitbake rev: 671aaab8cb7c494cd5c7621b45a6f41a203d8bb5)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than maintain data as part of the migrations (as was
done for the default project previously), create the default
(cli builds) project on demand as a by-product of getting
it from the database.
[YOCTO #8364]
(Bitbake rev: 5fd8e90ab9b81d1bd0d301bc1c91228ecbbea74b)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The buildinfohelper runs outside of Django, but still needs
access to our Django app classes.
Previously, the imports referenced the toaster.* app, which worked
fine. But in Django 1.8, this causes an error about the same
module being loaded multiple times from different paths.
Change the paths to our Django modules so they don't cause
this error to be thrown. We can do this as we've added our
application libraries to sys.path in the buildinfohelper anyway.
[YOCTO #8364]
(Bitbake rev: 070da64cf32c32b5ffc34d611b463c3a3960b419)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently BRBE is known to toasterui only when build is
started. It's passed to it with BuildStarted event. This is
too late as if build fails earilier than build starts toasterui
can not inform Toaster about the failure.
Set BRBE as soon as it's provided by Toaster.
This should make toasterui to be able to inform Toaster
about early build failures, e.g. failures during recipe parsing.
(Bitbake rev: d7819508dac488a64be3caec88db285cda9599ab)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If subprocess raises a CalledProcessError() error, e.g. from a call
like subprocess.check_call("false"), bitbake would try and pass the
object over IPC and fail, leading to an unusual error:
('__init__() takes at least 3 arguments (1 given)', <class 'subprocess.CalledProcessError'>, ())%
To avoid this, we turn the value into a string which prevents the
issues the IPC has trying to deal with the object (for the same reason
we deal with tracebacks here too).
[YOCTO #8752]
(Bitbake rev: 05695424b918fc81b16cbac70d79d8271a0b6045)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Toaster is not able to see ParseStarted and ParseProgress events
for command-line builds. This means it's not possible for Toaster
to detect failed builds, if the failure occurs at a point before
the BuildStarted event, as the build won't show up at all.
Add these events to the event mask, so that Toaster's toasterui
can detect and respond to them.
(Bitbake rev: 16bfd3e3d145705a2b3a05648ddbcacc7a338dfa)
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there are more tasks running than there are lines on the terminal, the data
scrolls in ways the UI wasn't designed for. This patch adjusts the UI just to
show the processes which fit onto the number of rows in the terminal window.
You can see the total number running from the counter in the top left as usual
and this makes warning and errors messages scrolling from the top of the window
work as designed.
Ultimately, scrolling would be nice but is for another time, this fixes the
biggest UI issue on highly parallel machines.
(Bitbake rev: 67b77658e2bfa849f6f55c9c262cb11d6bfdb399)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rename REGEX to UPSTREAM_CHECK_REGEX, REGEX_URI to UPSTREAM_CHECK_URI, and
GITTAGREGEX to UPSTREAM_CHECK_GITTAGREGEX to better reflect their purpose
and to reflect a common namespace.
(Bitbake rev: f0a9e783f9969573fd74edfa241ef14f18ac684e)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Removed buildinfohelper code which was trying to guess layer version of
the recipe using build request information. The code caused creation of
duplicated recipes as it resulted in layer version from layer index
instead of returning build layer version. As a result of this Toaster
UI was not showing any information about recipes.
Default approach used to find layer version seems to work much better as
it finds proper layer version. Now toaster will use it as the only
way to find layer version.
(Bitbake rev: 101690bda7ad55dc0657483233c90c374713755b)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When build information is the 'original' source of the information we
need to return the recipe that was created rather than the copy of the
recipe that is taken for keeping build history. We do this already for
command line triggered builds, but we also have this case for custom
images. We can simply check if the built_recipe exists instead of
special casing this.
(Bitbake rev: 9a8653bf602b2111dee7ee6a459682a68a695b22)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
buildinfohelper code expects only one HelpText object per
build/variable/description.
Current code creates more than one such an object, which causes
toastergui to crash with this exception:
MultipleObjectsReturned: get() returned more than one HelpText -- it returned 2!
Used git_or_create API to ensure that only one HelpText object is
created.
(Bitbake rev: e9b46803eb6f1f4044919abf90c8aeb3536e73ed)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Toaster needs bb server to be running all the time due
to merged analysis and managed modes. Server gets restarted
before every build triggered by UI, but it shouldn't be
terminated as it will influence command line builds.
[YOCTO #8279]
(Bitbake rev: e69c87c7842e796ffcd7193ecde22c8f688498f5)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Updated attributes of buildinfohelper object as they can
be changed for every build. For example brbe is set by
runbuilds for every build triggered by Toaster UI.
(Bitbake rev: ea3bc8d01704dc64f6cb7b4f5fe66c312a575174)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make the following improvements to edit_bblayers_conf():
* Support ~ in BBLAYERS entries
* Handle where BBLAYERS items are added over multiple lines with +=
instead of one single long item
Also add some comments documenting the function arguments and return
values as well as a set of bitbake-selftest tests.
(This function is used by the bitbake-layers add, remove and
layerindex-fetch subcommands, as well as devtool when adding the
workspace layer).
(Bitbake rev: e9a0858023c7671e30cc8ebb08496304b7f26b31)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you tried to delete the variable on the first line passed to
edit_metadata() this failed because the logic for trimming extra blank
lines didn't expect the list to be empty at that point - fix that bad
assumption.
(Bitbake rev: 8bce6fefdc5c046b916588962a2b429c0f648133)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For two reasons:
1) The important one: we hit the following bug when doing upstream version checks
on some webpages:
https://bugs.launchpad.net/beautifulsoup/+bug/1471755
2) Also, documentation for beautifulsoup states that memory usage and
speed is improved that way.
(Bitbake rev: 7546d4aeb3ba8fda9832081b84d93138dc5e58d6)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Our current OVERRIDES handling means we end up caching and checking for
a lot of possible override combinations which turn out to very unlikely.
A typical example is the SRC_URI variable where we have to check if
"URI" is an override. Having spent many hours working in this code, I've
realised all the actual overrides we use are lower case and our standard
variables are mostly uppercase.
This means we could gain quite some speed advantage if we write this
into the code, that overrides only consist of lowercase characters. This
patch shows how simple this is and the resulting speed gains are
significant. This is a significant change but tests show we don't appear
to have any users of capitals in overrides in any OE-Core metadata.
Before "time bitbake -p":
real 2m4.224s
user 7m32.312s
sys 0m7.116s
After "time bitbake -p":
real 1m26.009s
user 5m10.484s
sys 0m4.640s
This check could also be made conditional however I'm not seeing a need
to do that at present.
(Bitbake rev: c9b9443faa76ee7366b1400a56f826f3f9dec1be)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This looks reasonable until you realise self.localpath is a function. Data
expansion of something which isn't a string is the original value so this
code just wastes CPU cycles and makes no sense. Remove it.
(Bitbake rev: 37214ea9bf484998b75dbc1200d53f1afc5257ed)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Trying to expand a variable which isn't a string doesn't make sense.
(Bitbake rev: 62367cca1f1793eb9827406bcdd5980fdeb80a60)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Trying to expand a variable which isn't a string doesn't make sense.
(Bitbake rev: 54d0ddd166a6707b4f8c8535639e3055b783363b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The server state gets reset multiple times during startup and currently
we reload the codeparser cache each time. This is pointless and causes
unnecessary interaction time with bitbake.
(Bitbake rev: 4f700316933e8e7b2d27366e5ce6176895b913e7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than relying on bug 8411, which is conveniently creating
separate log files for each of our builds, create our own
log file for each build.
The log files are created in the same tmp directories that
bitbake users, but are timestamped to the millisecond to avoid
name collisions.
Each log file is opened on a ParseStarted event (for builds
triggered by Toaster) or BuildStarted event (for builds on the
command line: Toaster doesn't get the ParseStarted event
for command-line builds).
The log file is closed on the BuildCompleted event, or if the
build fails.
Because we start logging on ParseStarted for Toaster builds,
we're able to capture the "Build Configuration" section which
bitbake writes to output.
[YOCTO #8373]
(Bitbake rev: 7974203cd8bc66dff1fcc55f8723dedefaf72840)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Similarly to BB_TASK_NICE_LEVEL, add BB_TASK_IONICE_LEVEL which allows the ioprio
of tasks to be adjusted. This is in response to various qemu runtime timeouts
which have been witnessed on the autobuilder, seemingly due to IO starvation (we
already use NICE_LEVEL to adjust tasks). This has a fairly urgent need to deal
with certain 'random' failures we're seeing on the autobuilders in testing.
The format of the data in the variable is BB_TASK_IONICE_LEVEL = "<class>.<prio>".
For <class>, 2 is best effort (the default), 1 is real time and 3 is idle. You'd
need superuser privileges to use realtime. The <prio> value is a default of 4,
and can be set between 0 and 7 with 7 being lowest priority and 0 the highest.
The user can set this freely with normal privileges
Note that in order for this to take effect, you need the cfq scheduler selected
for the backing block device.
We could use nice wrapper functions for ioprio from modules like psutil however
that would complicate bitbake dependencies. This version has some magic numbers
but works on the main 32 and 64 bit x86 build architectures and can easily be
extended if ever needed. When we move to python 3.x, we can likely replace this
with standard calls.
(Bitbake rev: b9471ad147b102c45d65f5ffd9521864df7ff9c1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The console log data is written to is created at console initialisation
time and does not change over reset events. This ensures the
BB_CONSOLELOG value is correct over such resets by preserving it.
(Bitbake rev: 335eb2db228f7543a49de71f063ac72b865c947a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Metadata can define BB_CONSOLELOG as containing ${DATETIME} and
this can get expanded to a different value each time the variable
is read. In the case of BB_CONSOLELOG, this behaviour is not
desireable.
The values of DATE/TIME are locked down at build time but this is too
late for the purposes of ensuring the system can figure out the real
value of BB_CONSOLELOG.
The best way to do this is to set the variable into the datastore, thereby
preserving its value.
[YOCTO #8411]
(Bitbake rev: 021f2eb55ab5863b57ed1b3f19f1b329bc1ad477)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are some use cases where we want to read a variable but also
set the variable to the value read, effectively locking in any
expansion of it. This adds such a command.
(Bitbake rev: 0c0c524691e3d2ffd9953a106fcc06262cbde910)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Target objects are created before the build if build is
started from UI in build mode. However, in analysis mode Target
objects don't exist and need to be created using information
from bitbake events.
Added new API call get_or_create_targets to retrive existing
target objects or create them if they don't exist yet.
(Bitbake rev: ef69be31d133696bde54605f5a18da660099734c)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for the Subversion fetcher to checkout modules to a custom path
than the module name to avoid checkout is always module - svn is path
based and tag/branch-checkout might break builds because of invaid path specs.
(Bitbake rev: af88f538e61afa1b115be4d7afe00d8477f61750)
Signed-off-by: Jens Rehsack <sno@netbsd.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Additional config files passed to bitbake server with --read and
--postread options are rewritten by client bitbake even if
it doesn't use those options.
This is a show stopper for toaster as toaster command line
builds are based on the assumption that server is aware of
toster configs, provided by --postread option.
This behaviour is fixed by preserving values of --read and
--postread options when bitbake server starts and restoring
them if client bitbake doesn't explicitly specify them.
(Bitbake rev: 02c64f7487ca8ec5d32c440f5002c4b8f64b76a6)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
errno.ENOENT checked if deletion of the fullmirror fails.
Exception was thrown since module not imported
(Bitbake rev: d92ebfc34b69ad5df2d151e6b8299fbb5afa3e5f)
Signed-off-by: Logan Buchy <logan.buchy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BuildStarted event not fully represents build tasks for
the targets. If -c option is used to specify default task
it's not included into the event.
Made build targets to always look as <target>:do_<task>.
Consider default task (do_build or specified by -c command
line option) when normalizing.
(Bitbake rev: 0b0e214e6f53c97ad3d48f622c7fc0ca149956f6)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Critical errors (where a build failed for reasons of
misconfiguration, such as a machine being specified which is not
in a project's layers) were being ignored (only log records
up to ERROR level were being logged to Toaster's db). This meant that
the build would fail but would not correctly report why.
Add support for CRITICAL error levels to the LogMessage model,
include errors at this level in the errors property for a build,
and show errors at this level in the build dashboard.
[YOCTO #8320]
(Bitbake rev: b6eacbca9cacb607de864ab7d093deb296da8226)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we're building using toaster as just a listener to bitbake
(analysis mode) we need to handle the case where the toaster configuration data
isn't present so we don't need to try and update the existing information.
(Bitbake rev: a22faae2c3a5948356ce3cbc73c34509de65d370)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there are more packages listed as installed than we know about from
bitbake, and therefore have insufficient information to be able to
create a Toaster Package object then skip it. Also handle the case where
a dependency references such a package.
Also clarify the error logging.
(Bitbake rev: b4ce793685f70cab3f28cb4329aaaf3878cd62e8)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make sure we associate build data with the built recipe rather than
toaster's configuration copy of the recipe.
(Bitbake rev: 34d4ef7289d72d151ad0acdccab8b99c8c31221e)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Changed logging levels to more appropriate ones.
(Bitbake rev: 27d0360d13af0c698bf3a224b3f0d415f17bb678)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Added ReachableStamps event to the list of known events to
ignore. This should stop toaster throwing error message:
ERROR: Unknown event: <bb.event.ReachableStamps>
(Bitbake rev: cd4137e13af6964858640b78aa7fe6f1612be251)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you run bitbake-diffsigs against two differing sigdata files from
nostamp tasks it shows no difference despite the differing checksum.
Change the code so this shows up as a nostamp 'taint' and at least
makes the issue clearer to the end user.
(Bitbake rev: 97679d18955dadaa34f9450564e44da99984d140)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This firstly prints debug messages which show how bitbake decided to resolve
the virtual/xxx providers which is useful for debugging.
If the siggen has a tasks_resolved() method, it calls this, passing in
the mappings, allowing that to do things with the resolved names.
(Bitbake rev: d473fc84acddfd69a7207affcd89f65ea2ecf730)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When building an execution task graph, bitbake does resolve virtual/xxx
namespaces into specific providers. This data isn't exported anywhere
however.
This adds a function so that runqueue can at least retrieve this data
which can then be used by the system.
(Bitbake rev: ce51a51482d0900060512b24503714a730d72266)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BBPKGS is a confusing name from before we tried to straighten out our
terminology. Its also a mostly unknown variable that isn't in wide use.
I've been asked about it recently and before people start relying more
heavily on it, I'd like to rename it BBTARGETS which better describes
what it does. Its not currently in the manuals, I'd prefer to document
it under the better name. I've not provided any migration path for the
variable since I believe its unused currently.
It allows the targets to built to be specified from a conf file in
addition to those on the commandline.
(Bitbake rev: f60c6a2172bceeb5682dcb738a02c4bf26176566)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prevent edit_bblayers_conf() from adding layer(s) multiple times. This
happened when BBLAYERS variable was "listed" multiple times in
bblayer.conf - i.e. the configuration was split into multiple separate
assignments.
[YOCTO #8316]
(Bitbake rev: 5e423237f9f4ff7e7e03bf066b0142ba4bd82219)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This delete followed up the foreign keys and deleted things that were
not expected to be deleted.
This reverts commit 08000eb27e.
(Bitbake rev: 46b119eb62a5a612fe4c0847862d34f408e556f7)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create a copy of the built layer and the recipes associated with it.
This is so that the user can view the historical information about a
build. i.e. a snapshot of the layer version and artifacts produced at
that build.
(Bitbake rev: 0683d9a2b15e3234a94437aaebac84bfcca1420b)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously this layer relationship was done by trying to match path
information that came back to the buildinfohelper with trying to query for
data in toaster's layers table. This rarely matched due to the lose coupling.
(Bitbake rev: 838e77c7c3c4006abd1327343a004590ab652de9)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Otherwise the logger gets multiple handers (and the user get duplicate
logging output) if another tinfoil instance is initialized after one is
shut down().
(Bitbake rev: 74d67be7a4b591fab2278f7c184f282d11620c62)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The files-in-image.txt file is produced by bitbake after an
image is created, listing all the files in the image.
However, this list doesn't include the root directory ('/').
buildinfohelper.py then tries to construct the filesystem
tree from this file, assuming that every directory apart from
the root directory (which is special-cased) can be assigned
a parent. But because the root directory isn't listed in
files-in-image.txt, an object for the root directory is never
created.
The direct subdirectories of the root ('./bin', './usr' etc.)
then can't be assigned a parent directory, as the object
representing the root directory doesn't exist. This
results in a Target_File lookup error and causes the
directory listing page to fail.
Fix this by creating a fake entry for the root directory
in the Target_File table, so that the direct subdirectories
of / can be assigned a parent. Note that it doesn't matter
that the root is faked, as its properties are never shown
in the directory structure tree.
[YOCTO #8280]
(Bitbake rev: a4015768183e5a3fa39a6c2b4dea0088ca182d80)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Due to re-creating Target objects from bitbake events task information
stored in original objects is lost.
There is no valid reason to remove existing objects. It's safer to query
them instead of re-creating as original object contain more information
than events coming from bitbake.
(Bitbake rev: aab4aff75eefb31aa53885d7735feee5daa294aa)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently this module is dereferencing errno.ENOENT but the python module "errno"
is not imported, which causes a crash when fetching from a git repository.
(Bitbake rev: 93e4c9bb2393b1074f5a01e7eaaac742a59d8086)
Signed-off-by: Romain Perier <romain.perier@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The dependencies of OVERRIDES were not including DEFAULTTUNE in OE-Core.
This is pulled in by a bb.utils.contains() reference which the override
dependency tracking code wasn't accounting for.
This patch ensures we do track contains references too.
(Bitbake rev: f3ee534cb0560dbb5f88a0ffe01e9305bae102e1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Right now, OVERRIDES dependency variables set using ??=, e.g. TARGET_ARCH
in OE-Core don't have their dependencies tracked. This is a bug, fix it.
(Bitbake rev: 944734503768f9e9223ef041f2d7873455418a54)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sadly its not enough to consider the dependencies of OVERRIDES, we
need to resolve their dependencies and so on recursively. If we don't
do this, some variable can be changed and the resulting data store is
incorrect.
(Bitbake rev: 82143ac064d391300e762ba7520ef1f8df18b574)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the values that make up OVERRIDES are themselves overridden,
we end up into some horrible circular logic. Unfortunately some
metadata does depend on this functionality.
e.g:
DEFAULTTUNE_virtclass-multilib-xxx = Y
which changes TUNE_ARCH
which changes TARGET_ARCH
which changes OVERRIDES
As a solution, we iterate override expansion until the values don't
change. If we iterate more than 5 times we abort and tell the user to
report the issue.
(Bitbake rev: 10279697c701e01bf6fdd5e9f92792ef5134807b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were maintaining state in the form of ud.repochanged to determine whether
we need to write the tarball in the case where the tarball already exists, but
this state is maintained only in memory. If we need to update the git repo,
set ud.repochanged to True, and then are interrupted or killed, the tarball
will then be out of date.
Rather than maintaining this state, simply remove the out of date tarball when
we update the git repo, and it will be re-created with updated content in
build_mirror_data.
[YOCTO #6366]
(Bitbake rev: eaaa81393f181432c8586b17ade623f42c9fed2e)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of reproducessing the same line over and over and over, we remove the
current line from the mirror list. This permits us to re-evaluate the list
while excluding all matches that have previousily occured.
Without this fix, adding this test results in a failure:
RuntimeError: maximum recursion depth exceeded in cmp
(Bitbake rev: 24a8e9a5b0ba145ae589178d74365c986ebca325)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The BB_CONSOLELOG variable changes by the time we read it in
BuildInfoHelper. This means that the log file location we
are using is incorrect, so the links to the cooker logs don't
work.
Instead, read it at the point when the BuildStarted event occurs
in toasterui. The BB_CONSOLELOG variable has the correct value
here, so pass that to BuildInfoHelper.
[YOCTO #8209]
(Bitbake rev: 20609eebee0d2318806cf81913e7ce6dc1005507)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
No code changes, just refactoring to allow for functionality
changes by moving things to a separate function.
(Bitbake rev: 2eb934814179ccf42e3d424dabe26b17d013a7ed)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there is a space in a directory name containing a file in file-checksums
(e.g. from a file:// url), you currently get tracebacks from bitbake. This
improves the code to handle colons and spaces in the file-checksums names
since it possible to figure out the correct names.
[YOCTO #8267]
(Bitbake rev: 87282b283921a58426f24fb21151db457c5bca66)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a new bb.utils.signal_on_parent_exit() function so that a process
can register to recieve a signal when the parent dies. There is no
POSIX standard for this and the implementation is Linux specific.
Alternatives would be having an open pipe or polling os.getppid()
for changes but this seems more effective and less invasive to most
of bitbake's code structure.
We need to be able to determine when parents die to ensure child
processes stop running in a variety of circumstances to avoid
locks being held and ensure clean shutdown.
Roughly based on https://gist.github.com/evansd/2346614
(Bitbake rev: 34974f5e30e9b09c016481e4c81c156a5f379784)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if you send a SIGTERM to the bitbake UI process, the system basically
hangs if tasks are executing. This is because the server process doesn't
actually try any kind of shutdown before exiting.
This patch trys executing a stateForceShutdown command first, which is
enough to stop any active tasks before the system exits.
I also noticed that terminate can execute multiple times, once at SIGTERM
from the handler and once from the real exit. Double execution leads to
stack traces and potential hangs (writes to dead pipes), so ensure
the code only can run once.
With these fixes, bitbake much more correctly deals with SIGTERM to the
UI process.
(Bitbake rev: 1032ddddbe3241da02ebb3608a1c40f9123b9e80)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When our clone exists, but is out of date, and the attempt to update it fails,
we don't necessarily want to remove the entire clone, particularly if it's
a large repository.
(Bitbake rev: 19af272ba5256653edeff6acbceeb09e3e478d61)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the main fetch method doesn't support checksums, the user will not be
defining them in the recipe, so we don't want to check them for
premirrors/mirrors either. This ensures that we never error due to missing
checksums on a git mirror tarball.
(Bitbake rev: 24c79bbed361b37f12d3351af13602e3d4386f4c)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using an absolute file URI, there's no host, and the path starts with
'/', the dir under ${DL_DIR}/git2/ ends up starting with '.', so is hidden.
Remove any leading '.' to fix this.
(Bitbake rev: 8dce6964d56b36a77fb113f2ad496cc992a5ff36)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Show the user only the portion of the traceback which was from the metadata,
nothing from bitbake's internal calls.
(Bitbake rev: c45054aef03393fa0bf70e853ddcfc55988493cf)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This gives us the needed context of the original ExpansionError, which is
invaluable when we have a chain of function calls in the expansion.
(Bitbake rev: c514b6fbea77ede1b7871b89592a33ed39b1d71c)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The output of "bitbake, -c listtasks pkg" lists tasks with their real names
(starting with "do_"), but then "bitbake -c do_task" fails, as "do_" always
gets unconditionally prepended to task names. This patch handles this error
by checking whether a task starts with "do_" prior to prepending it with it
when the task runlist is being constructed (and a few other corner cases).
[YOCTO #7818]
(Bitbake rev: dd3050ceef37ac556546e940aa596ce96ef6c8df)
Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you start and suspend a bitbake execution so the bitbake lock is held,
then try and run "bitbake -w '' X", you will see bitbake return an error exit
code but print no message about what happened at all.
The reason is that the -w option creates a "UI" which swallows the messages. The
code which handles this exit failure mode thinks a UI has printed the messages
and therefore doesn't do so.
This adds in an extra parameter to the UI registration code so that we
can figure out whether its a primary UI or not and base decisions on whether
to display information on that instead. This fixes the error shown above and
some bizarre failures on the Yocto Project Autobuilder.
[YOCTO #8239]
(Bitbake rev: d1d60a68c2de40c2984d5040d14251c1be121b0b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake utils' movefile is now prone to malform the destination
file with duplicated file name strings. Fixing it to force a file
name append iff the dest argument is a dir not a file name
(Bitbake rev: 38dd27f7191da002a16c561be3790ce487045b01)
Signed-off-by: Benjamin Esquivel <benjamin.esquivel@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Command line builds are associated with a "default project"
(as we currently require a build to have a project). This
acts as a container for builds initiated outside Toaster.
Currently, this project is marked as the default by its ID
being 0. However, this doesn't work with MySQL, as MySQL
won't allow 0 in a foreign key which references an
autoincrement field.
Instead, use an is_default field to track the default Project
for builds initiated outside Toaster.
Add a method to fetch this default project, rather than fetching
a project with a magic ID.
Add this default project in a migration, rather than as a side
effect of a get_or_create() style method.
Also ensure that builds always have a project explicitly assigned
to avoid any magic with a build's project foreign key defaulting to
0 (as it no longer does).
[YOCTO #7932]
(Bitbake rev: 71b709a1bbc26d89d61873763b467d21e625b274)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When moving a file via the os.rename function, it was missing the
destination file name which caused an OSError
[YOCTO#8180]
(Bitbake rev: b147ba0341d87e077bd2b09ef4355976ecd2d26b)
Signed-off-by: Benjamin Esquivel <benjamin.esquivel@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We want addtask to be able to bring back a deleted task, but we don't want its
previous dependencies to come back with it, so rather than marking a task as
deleted and then skipping tasks marked as such, actually delete the task and
its dependency information in deltask.
While we're in that part of the code, also fix a couple 'not foo in bar'
instances to 'foo not in bar', which is preferred in python.
(Bitbake rev: 94b3f3d6bdfbfa47f7eb3c3de64940a145b2ddd1)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow any listeners for this event (such as buildhistory.bbclass in
OpenEmbedded) to find out if the build was interrupted rather than
completing normally. The value will be 0 if not interrupted, 1 if
interrupted waiting for remaining tasks to complete, or 2 if force
interrupted (stopping any running tasks immediately).
(Bitbake rev: df2b778efd2ecc48f6c5a3ed446f6459f2250035)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're importing the server and UI modules in order to check they are
valid, but it turns out that that has some nasty side-effects. We don't
actually need to do this except when --help is passed or the module
doesn't exist, so rearrange the code so that we only do the module
listing in those two cases.
Additionally, let's just go ahead and catch all errors on import; we
really don't care to have them cause a failure now.
(Bitbake rev: c9dc6d9c86e8b887821a6d00346bd0b09e1da97c)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This exception was triggered in toaster from recent changes and is completely
breaking the whole of bitbake. Add the exception to the list so at least only
toaster is affected.
(Bitbake rev: f64def7cb6069dc1134fcd546bb59e4030c7376f)
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch fixes pylint issues in the toaster build data logger,
toasterui. The following types of warnings are touched here:
* fixing imports
* unused variables are set to _
* logger calls now use lazy evaluation instead of formatting
the string
* correct whitespace identation
* removes unneeded "pass" statements, and extra parantheses
* disable specific pylint warnings when decideing to override
them
(Bitbake rev: 947d47f15048baa967f88e03d80014e88ce152aa)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prompted by issues discovered during running pylint on
the toaster sources, this patch fixes:
* missing import in toaster ui
* improper call of function in toaster ui (logger.debug)
* improper function definitions in bbcontroller
* invalid references to objects in bldcontrol.models
* proper initialization of object fields in ToasterTables
Also inhibiting specific pylint errors in files where
the problems are mis-identified.
(Bitbake rev: 1c71955c416fb68455f7f70669aba4202c411807)
Signed-off-by: Alexandru Damian <alexandru.damian@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake-layers needs to map recipe and class files to the layer they
came from within the show-recipes and show-overlayed commands. However,
it turns out that mapping a file to the layer it came from is not as
trivial as it might seem. To do it properly we need to match the path to
an entry in BBFILES then map that to the collection name using
BBFILE_PATTERN, then map that to the actual layer using variable
history. If it doesn't match any entry in BBFILES, then we can fall back
to BBFILE_PATTERN (to handle classes and conf files).
This fixes the layer name not showing up properly in the output of the
show-recipes and show-overlayed commands for recipes in layers such as
meta-intel that have subdirectories in BBFILE_PATTERN. It also fixes the
priority not showing up in show-layers for such layers.
As part of this I've added a function to VariableHistory which for a
space-separated list variable gives you a dict mapping the items added
to the files in which they were added. I've also fixed
bb.utils.get_file_layer() and reduced some of the duplication by using
this function in bitbake-layers. Also fixes the priority not showing up
for layers such as meta-intel
Fixes [YOCTO #8160].
(Bitbake rev: e852f6cabd7489585477ab567a1afeb2252377ac)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Provide us with a means of showing the list of UIs / server choices for
the command line help, and do the processing in one place for both.
(Bitbake rev: 24035c1daad5a904c3372d21d44191ee8182338f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a file no longer exists, drop it from the cache silently instead of
generating a traceback. This was visible in some cases when a recipe was
deleted when bitbake was resident in memory.
(Bitbake rev: fe105b9042bdac4afd9f38fcf92bfdc2c04ec23f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now we have the bbappends list and all users have been converted over to
use it, we don't need this anymore.
(Bitbake rev: 279770c42d4c63aa2cebce331b55a92a564b50ac)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst collecting this list on the fly may be quicker, doing so within a
function that's meant to *query* the list of bbappends is poor practice.
(Bitbake rev: 5c12aa3b0010d7d1733e54a0ca7d0af465454210)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we can't connect to the server we should error out, because it might
not be that the server is actually dead - it might just be unable to
execute commands.
(Bitbake rev: d4b921676859d6ba4e1922fa4682ee941652f483)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The prefile/postfile options weren't working in memory resident mode
because they weren't being passed through to the server, so ensure that
they do get passed through and that the server is reset when the values
come through.
(Bitbake rev: a3f7dc042fc7b1c308bfd248431930eb8ba50326)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a means of ensuring that synchronous commands that read the results
of the configuration trigger a reparse of the configuration if any
underlying files have changed.
(Bitbake rev: aaf3cc024315450c1674819edf2a4e5cdf293f35)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Unfortunately we were acting on inotify notifications about any files
changing within the watched directories, not just the ones we actually
care about. In OE the build directory is in BBPATH and hence it gets
watched, and we write things like bitbake.lock and
bitbake-cookerdaemon.log to that directory, hence effectively
notifications were being tripped on every bitbake invocation. To avoid
this, record which file/subdirectory we're interested in against each
watched directory so we can ignore any events for files/subdirectories
we don't care about.
Additionally, if we move up to the parent dir, ensure we haven't already
seen it before adding a watch on it (we were previously calling
watcher.add_watch() on the same directory multiple times before in a
typical OE configuration).
(Bitbake rev: bc39b8da34c046b629c43fd0a8cac2efbf1c060f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If sys.exit() is called within a command run over XMLRPC, the XMLRPC
server is effectively trashed (apparently listening but no longer able
to respond to commands). We need to intercept the SystemExit exception
and deal with it as we would any other exception.
(Bitbake rev: 95e391acbc3b4efd6c77637a1ce815012ae0f09b)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With python you should not assign a list as the default value of a
function parameter - because a list is mutable, the result will be that
the first time a value is passed it will actually modify the default.
Reference:
http://docs.python-guide.org/en/latest/writing/gotchas/#mutable-default-arguments
(Bitbake rev: 7859f7388f2e3f675d0e1527cfde18625f36f637)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently bitbake is the only one that knows the relation
between PN and PROVIDES. In some cases it is needed to have
this relation in the data store (the bootloader it's a good
case).
This adds the PROVIDES to the taskdata, so it would be in
the data store as one field of BB_TASKDEPDATA.
(Bitbake rev: a660787311d2961c66c0443bf0e2e094c9baef1b)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you load memory resident bitbake, then change bblayers.conf or other
configuration files, then execute your first command, bitbake will not
notice the changes. This is because it adds the file watches during
inital parsing, which happens at the time the first command is run.
To fix this, we move the addition of the file watches to earlier in
the process, so bitbake then does track them correctly. This also
avoids some issues Paul was seeing with tinfoil2.
(Bitbake rev: b9375c73e736003e66575969c9ea244403e47aeb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've had a long-standing bug where a legitimate error reading a file (IOError
or OSError) is always suppressed as though it was a 'file not found' case. As
a concrete example, if you do a `chmod 000 conf/local.conf`, it'll silently
not parse local.conf, rather than erroring to let the user know about the
problem.
Fix this by handling the ENOENT case specifically.
(Bitbake rev: e691312a3add222b04e7b2f52f8df6abcb9068bf)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tracebacks are of extremely limited usefulness in this context. The exceptions
carry the necessary context already, and the user doesn't care about the calls
in bitbake internals that led to an expansion or parse failure.
(Bitbake rev: 9b95fa94eaae452ac7814f1e67c2f7a6314c52f1)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Set logging level for the Toaster logger to get messages
to toaster_ui.log
Reverted previous workaround.
[YOCTO: #7965]
(Bitbake rev: f1d8e0ff7233f17cff3a56db10d2443b392b407d)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
[\.-_] means "any character between . and _"
What was meant here is certainly "any character from the three characters .-_"
(Bitbake rev: af13eaba627f199f91c048c435b9dbe19c79527f)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the PR server or indeed any other child process takes some time to
exit (which it sometimes does when saving its database), it can end up
holding bitbake.lock after the UI exits, which led to errors if you ran
bitbake commands successively - we saw this when running the PR server
oe-selftest tests in OE-Core. The recent attempt to fix this wasn't
quite right and ended up breaking memory resident bitbake. This time we
close the lock file when cooker shuts down (inside the UI process)
instead of unlocking it, and this is done in the cooker code rather than
the actual UI code so it doesn't matter which UI is in use. Additionally
we report that we're waiting for the lock to be released, using lsof or
fuser if available to list the processes with the lock open.
The 'magic' in the locking is due to all spawned subprocesses of bitbake
holding an open file descriptor to the bitbake.lock. It is automatically
unlocked when all those fds close the file (as all the processes terminate).
We close the UI copy of the lock explicitly, then close the server process
copy, any remaining open copy is therefore some proess exiting.
(The reproducer for the problem is to set PRSERV_HOST = "localhost:0"
and add a call to time.sleep(20) after self.server_close() in
lib/prserv/serv.py, then run "bitbake -p; bitbake -p" ).
Cleanup work done by Paul Eggleton <paul.eggleton@linux.intel.com>.
This reverts bitbake commit 69ecd15aece54753154950c55d7af42f85ad8606 and
e97a9f1528d77503b5c93e48e3de9933fbb9f3cd.
(Bitbake rev: a29780bd43f74b7326fe788dbd65177b86806fcf)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since shell scripts run with 'set -e' the final exit at the end of the
script can only be returning 0. However, for correctness and to follow
the original intention of the 'cleanup' commands, let's fix the typo
and return the success of the last shell function rather than the
success of unhooking the exit trap handler.
(Bitbake rev: bef724057f1ea81571dd3ac5a9cf862f818434b5)
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Calling record() for each override alteration is slow. Since we now expand
overrides dynamically we don't have to record the log data at each alteration,
we can instead print it directly from the existing data stores at variable
history print time using the exact same data stores.
This massively improves performance of the data store when parsing
with bitbake -e for example, it will improve memory overhead as well.
The only downside is that VariableHistory has to poke into the datastore
for some of its data but that seems an acceptable tradeoff rather than
double caching.
(Bitbake rev: 100b447a161ef20fa559e39516cd32fa78e38262)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As things stand now, bitbake -e (which turns on all the caller tracking)
of OE-Core generates around 9.5 million stat calls which is slow and the
largest single thing on the profile data.
This is because infer_caller_details() calls traceback.extract_stack()
which adds line contents to the traceback. This in turn calls python's
internal linecache code which calls stat on every file for every callback.
We don't even use that info. We only even want a single frame of the stack.
Instead, open code for the pieces of information we need. Also, only
obtain the stack once for both halves of the infer_caller_details()
code.
This reduces the number of stat calls to around 0.5 million and significantly
improves parsing with bitbake -e.
(Bitbake rev: 7be76d8f79ea92fd4bd36146eb9a4b86551b526d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We should use the d.xxxVar syntax rather than the older function style.
Also replace 0/1 with the more pythonic True/False.
No functionality changes.
(Bitbake rev: 90fdd69cca951f8bd2ff634f3b42fccd4fc03095)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Minor formatting improvement in generated shell scripts.
Add a newline after the last export emitted by emit_func. Also, remove
chance of putting an extra newline in the middle of the exports, since
we never want to do that.
(Bitbake rev: 5117d9b5d32d5d81adf70fa3e3feac9cef654240)
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
One drawback to tinfoil is that once a cooker is created, it will hold
the cooker lock and stop any other bitbake execution against a directory.
Add a shutdown method to tinfoil, allowing other users to use
the build directory after the tinfoil usage is no longer needed.
(Bitbake rev: e44ce85fe551677fc0dcc1da4f789a0c13093ff1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
One data store changing a variable poked through into a different data
store. This test case replicates that issue where the value 'B' would
become unset/disappear.
We also enhance parsehelper to generate files with an optional suffix
such as bbclass.
(Bitbake rev: 5c4179f58a4e04f1c354df5f17d1860eb403f0ac)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using deepcopy() caused a major performance regression. Turns
out we can work with a shallow copy as long as we force the
recreation of any list that we change, effectively a poor mans
CoW. This isn't too invasive and avoids the huge overhead of
deepcopy so this seems like the best way to have performance and
a working data store.
(Bitbake rev: 32dff4749c32f32e947c42c86f8357b8b607354b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Change the test name to be more specific on what is tested.
(Bitbake rev: 3e39156bc330c4c726058a5b9c13d33e2daad89f)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now latest_versionstring method returns (version, revision) for comply
the new return convention needed by SCM's like git get the current
revision.
bb/tests/fetch.py: Updated wget latest_versionstring test for comply new
convention.
[YOCTO #7605]
(Bitbake rev: 8d454646cbe1b04758ca178d8c6fcfd02b818b7b)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to know the revision associated with the upstream version in
SCM like git, this change broke return convention, oe-core recipeutils
get_recipe_upstream_version need to be updated.
tests/fetch.py: Updated git latest_versionstring test for comply new
convention.
[YOCTO #7605]
(Bitbake rev: fd175dc90024c503134c11cbd83e77d88c406ac8)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Don't limit the tag search for only tags end with ^{}.
(Bitbake rev: 7006ab313766344cf33481228465082ed5977d28)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
dbus sets RDEPENDS_dbus-ptest_class-target = "X". nativesdk as
BBCLASSEXTEND was expanding and setting various RDEPENDS variables
which were clearing overrides queued against RDEPENDS. The problem
was that this was leaking into other data contexts such as the
target case. This was because overridedata was a shallow copy.
Replacing the shallow copy with a deep copy avoids this problem of
leakage between the data stores at a small performance penalty.
(Bitbake rev: 83e2923cd3c8f2e25987132f85b06ce72bc941d7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The recent change to connect through the shell logging functions had an
unexpected side-effect - bb.error() and bb.fatal() cause a flag to be
set internally such that BitBake's UI will not print the full task log
on failure; unfortunately we have in places within the OpenEmbedded
metadata called these shell logging functions under error situations
where we still want to see the full log (i.e., the message we're sending
doesn't include the full error). Thus, provide a mechanism to fatally
exit with an error but unset the flag, utilising the built-in python
logging functionality that allows extra values to be passed in the log
record.
(Bitbake rev: e561b997c55e8537d82aa1339adfff4505cc38b7)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're calling select() to find the fds that have data available, so we
really ought to check the return to see if the extra file is in there
before trying to read from it. This is part of the fix for the
performance regression that this code introduced (5-10 minutes extra in
a reasonable size OE build); the rest is down to an issue in the way
that pseudo currently handles FIFOs and will need to be addressed there,
see: https://bugzilla.yoctoproject.org/show_bug.cgi?id=7993
Solution suggested by Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 3e2db8d7eaa0f14813213d1c95d3ee9efd97dc9d)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a variety of tests which were found to be useful when working
on the data store recently.
(Bitbake rev: 5c5f8da509f6bbc1fad263462142519ef3d54a35)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that overrides get expanded 'on the fly', change appendVar
and prependVar to work using _append and _prepend, else we'd have
to re-implement pieces of getVar and the timing of expansions
becomes problematic.
Using _append/_prepend equivalence gives the behaviour users likley
expect from these functions.
(Bitbake rev: 40d661aaf7a563c6447b073310c5f2fdae6ca3d0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The expand cache can be cleared in the middle of getVar now
due to the use of operations like delVar. We therefore need
to check if variables are in the cache before accessing in
case it was cleared.
(Bitbake rev: a97ce216cfe44136f742383542954bfce027831e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
d.keys() was not returning items which had no base content but the
variable came into existance through overrides. We have to process
self.overridedata to find these other variables.
(Bitbake rev: 77fc57c610d7f65f7e328ad555f525b2f8bdf1d7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This performs better than continually regeneratiing the set().
Also only use set comparisions when its necessary to save some overhead
as issubset() is slower in the single item case.
(Bitbake rev: db9a4eedcf78969ec50bf0e4a8defba8ff0daa4a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When expanding OVERRIDES, its possible someone might try and override a variable
that is used in OVERRIDES. This could lead to infinite recursion. Add in
guards against this.
(Bitbake rev: 07d773369f571028c2cf82dd1f65d9731af6d00e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Updating the value of OVERRIDES whenever it changes turns out to be
extremely expensve/pointless. Instead, clear its value and re-establish
the value when we're going to use it.
This gives significant speed back.
(Bitbake rev: 41cf8d0c92d2d8a33fdad0921e424a0024914be1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using an internal flag for override mapping turns out to be slower
than is optimal, not least as we don't want the keys list to list
variables that have no value other than a potential override expansion.
Maintinaing a seperate cache is therefore more optimal from a speed
perspective.
(Bitbake rev: 1a39ccf66b30b54e1274669cb0e1dc8fc72dba49)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Trying to look up a variable called 'copy' in COW is problematic due
to internal implmentation details, at least avoid tracebacks from this.
Also don't duplicate override history (which is an atrefact of changed
override behaviour) as otherwise the bitbake -e output is convoluted.
(Bitbake rev: dddff5b7b8e6c18515b43389cef35503468b843d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When parsing we find problems if we clear prepends/appends when
setting variables during the initial parsing phases. Later, we actively
want to do this (in what would be post finalisation previously).
To handle this, pass a parsing flag to the operations to control
the correct behaviour for the context.
(Bitbake rev: ae87f5b8bf16191b3201cfb445062938eab992a0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Compeltely remove the replaces functionality and move all overrides
handling to getVar time. We can move the cookie cache into a hidden
flag within the variables themselves.
This removes the need for any of the internal_finalize steps.
This obsolete the need for the _seen_overrides COW cache.
(Bitbake rev: 2a0b73110bede91777ada54d1d89b45fb6034b6c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than repeatedly expanding the value of OVERRIDES, cache
the value and update it when things change.
There were also some bugs introduced in the replaces functionality
which this approach fixes by ensuring the replaces data is updated
at the correct points.
(Bitbake rev: f3b7c3e054ce230ea5c2db5813390383e8dfd6db)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These write operations should clear the expand cache since they can
influence returned variable values but currently do not. Fix this.
(Bitbake rev: a075332c9e13be35f1ae84adc6b29e9137a487ff)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than the heavy lifting in internal_finalize, move the bulk of
the functionality to getVar and rely on a new internal replaces variable
to keep track of any mappings that are needed. This removes the need
for the COW _special_values cache.
This change in functionality also implies we need to track any changes
not only to OVERRIDES but also any variable which OVERIDES depends upon.
Add code to handle this.
Explicitly list FILE as a value OVERRIDES depends upon since it doesn't
automatically get detected and is of key importance to OVERRIDES,
otherwise PN doesn't update when FILE changes.
(Bitbake rev: a6f1377ce3386d274882072d1ae6da3b1834149b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move the update_data functionality into internal data store operations
so the main finalize (update_data) call is a nop.
To make this work we need to call the internal finalization function
whenever OVERRIDES is changed to ensure values get updated correctly.
This has performance issues but the subsequant patches look into this.
(Bitbake rev: 546d9932d6c4413824319a9d780c0d77d2725f4a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since Python 2.7.9 ssl cert validation is enabled by default
see PEP-0476, this causes verification errors on some https
servers so disable by default.
(Bitbake rev: e177170200ece76b36e3f7d5597651fdef67736f)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Only use ProxyHandler opener when exists proxies in env.
(Bitbake rev: 0f062c5d99e12ce20c0e46a8e602448032144ff6)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2/__init__.py: Add connection_cache param in Fetch __init__.
In order to pass connection cache object to checkstatus method.
[YOCTO #7796]
(Bitbake rev: 9fa6407e6cefe66c77467419a8040d6957a6bb01)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Set BatchMode=yes instead of PasswordAuthentication=no. This will make
sftp fail immediately, not only when SSH requires interactive
authentication, but also on errors related to host key verifcation.
(Bitbake rev: 31305853a177735cc9c4553ea8905cd0acfcb100)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Create a branch and named as upstream branch when checkout source
* Set the branch to track remote branch.
(Bitbake rev: 1ba20e4fe9c884515b200589fe379ad5eeda10bd)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you have a situation where you are getting a "Nothing PROVIDES" error
(for example when something you request to build DEPENDS on something
that has been skipped or doesn't exist) it would be useful to be able to
use bitbake -g or bitbake -e to debug it, but currently both of those
are blocked by the error.
This patch adds an "allowincomplete" option to taskdata and uses this
for the -e/-g bitbake options. The NoProvider errors are still
printed and bitbake does return an error exist code but the environment
and task graph files are generated.
[YOCTO #7623]
(Bitbake rev: 2ca36a9f088438a8d1db44119c704f9480b04298)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create a fifo under ${T} for each running task that can be used by the
task to send events back to BitBake. The purpose of this is to allow
OpenEmbedded's logging.bbclass (which provides shell function
equivalents for bb.warn(), bb.note() etc.) to have those functions
trigger the appropriate events within BitBake so that messages are shown
through the BitBake UI rather than just in the task log; to do that we
just call the python functions.
Part of the fix for [YOCTO #5275].
(Bitbake rev: bae330a1f8a59a912eca71177b3c6ba7c92a2f38)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to pass connection cache object to checkstatus function
add fetch parameter.
(Bitbake rev: fbb9c6f5538084e125b58118a86968908e6f895b)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
FetchConnectionCache class acts as a container for socket connections
useful when implement connection cache into fetcher modules.
(Bitbake rev: 454da2cd17539ceb9caad6d76f034757e44ee12f)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before exiting the UI, unlocks the bitbake.lock owned by cooker; this
way consecutive bitbake executions can lock it again without trouble.
[Yocto #7941]
(Bitbake rev: 69ecd15aece54753154950c55d7af42f85ad8606)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Function which calls cooker's unlock method, which in turn unlocks bitbake.lock
file.
(Bitbake rev: e97a9f1528d77503b5c93e48e3de9933fbb9f3cd)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, once set, DATE and TIME in the metadata remain unchanged.
This is suboptimal for cases where cooker is reused. This ties setting
the variables into the BuildStarted event which seems like a more
appropriate time to do so.
It also changes BUILDNAME to be based off DATE/TIME by default if not
already set so that the data is more consistent. We therefore need to
expand the value rather than the previous default of not doing so.
This change does mean the date/time values are in sync across all
variables too.
It does mean bitbake now has special knowledge of DATE/TIME but that
would seen unavoidable (other than doing this in event handlers which
has its own set of downsides).
[YOCTO #5187]
(Bitbake rev: f883cf240266ee7be2cbd8971a8164cf4df9e372)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The incredibly useful recipe:do_task syntax on the command line isn't documented
at all. This isn't much but it's better than nothing.
(Bitbake rev: 7f4c07886ecff4ac77fdd2165bedd179099fcf19)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is nessary when specified branch with submodules is different then
default (master) branch.
[YOCTO #7771]
(Bitbake rev: f7b0b5e33e00f3ce0744322eee93835ee76bf184)
Signed-off-by: Felipe F. Tonello <eu@felipetonello.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make sure the error value is initialised and update the string match
that we have to identify bitbake's current startup status.
Patch contribution from Eduard Bartosh
(Bitbake rev: 915ba08a8a3013e9787e564f2ffd8698c948f433)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These aid when debugging parts of the codeparser cache since the
object contents becomes identifiable.
(Bitbake rev: 344b098c7eafc2bcc5c6b44ea47985bc0cb446b5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The main parser and other code copes with empty python functions but
the python codeparser does not. Fix this to avoid errors with code like:
python do_build () {
}
which currently results in the obtuse:
"Failure expanding variable do_build: IndexError: string index out of range"
[YOCTO #7829]
(Bitbake rev: e4f594c670189e04d58ce7d160fc1d86123620af)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
* do_unpack error:
abort: repository DL_DIR/hg/vim.googlecode.com/hg/vim not found!
* The mirror tarball doesn't work
- Add the build_mirror_data to create the tarball
- Unpack the mirror tarball when needed
* The hg files will put in the dir like git: DL_DIR/hg, it was
DL_DIR/hg/path/to/src/uri/path in the past.
(Bitbake rev: 378647281ae883dd726f1e1b775dd35ef6a0e8d1)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fetching directories with p4 worked but single files did not. This
patch from Helmut Auer (helmut.auer@harman.com) fixes that issue.
[YOCTO #7891]
(Bitbake rev: 39da6579901c62a83ed9319c2016c58fbbc108fe)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Events can call each other recursively, e.g. an event handler can call
bb.note which in turn generates another event. If these loop, it
can lead to multiple deletions of 'd' from __builtins__ which
can fail since __builtins__ is global scope.
Add handling to only remove 'd' when we added it and it wasn't already
present.
(Bitbake rev: b45952650ce8f470f124df36185b79e0d3a1783a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some handlers hook on BuildComplete so it avoids certain event races
to finish the command after the BuildComplete event is sent out.
This means the UI is available to handle events until the command
completes.
What appears to be a race on one of the sanity tests for event handlers
triggered this change although the failure is hard to reproduce.
[YOCTO #7921]
(Bitbake rev: e42d7c47a06fbb5981e0313478c8e3656b99f4e7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch verifies that BRBE is set before trying to use
it to read the checkout paths. This is needed for builds
ran outside Toaster control.
(Bitbake rev: e04807cd3135c9de96cc7f79245f329c24618b85)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We remove the "timespent", "errors_no" and "warnings_no" fields
in favor of computing the needed values at runtime. This prevents
inconsistencies in the UI.
Also removeing all references to BuildRequests from the interface -
all build details now display in the build dashboard.
Minor fixes related to data logging.
(Bitbake rev: 44f37394ed3e4ca02f940be172fe4395b0ee0f7d)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a set of fixes that repair the interface after
we switched from displaying BuildRequest data to Build data
in the formerly "managed" mode.
(Bitbake rev: 57f790b0c56297af8c83d5def8461bd5d61fe4af)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds logic to complete changing the interface
from showing BuildRequests to showing Build data.
The BuildRequest data is now transformed in Build data with
proper Toaster exceptions being recorded instead of listing
problems during startup as build errors.
(Bitbake rev: 51a41172d0b390370f9a38696b1ac65666ada4d2)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake does really weird things with negative or zero numbers of threads
which is confusing to the user. Add a sanity check for this.
When you have code doing arithmetic on the values and a VM reconfigures
to only a single thread, negative numbers are easier than you'd think.
(Bitbake rev: 32166ac3c85ff3c04081580ae76bd63590d6ff3e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we see a BBHandledException in runqueue, the understanding is the system
handled it, printing a log and traceback is just confusing.
Therefore only print these in the cases where its an unknown/unhandled
exception.
(Bitbake rev: 29d28e22ce431c3d3aabdb88ff4d8cca67a1cfad)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we see a BBHandledException in the idle handler, the understanding
is the system handled it, printing a log and traceback is just confusing.
Therefore only print these in the cases where its an unknown/unhandled
exception.
(Bitbake rev: d88ecc2bc44dce8fd92ca3a2c0fd4124a5e464fa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if the fakeworker failes to start the output from bitbake is confusing.
Improve the error handling to give a clear indication of what failed.
Patch from Chris Larson.
(Bitbake rev: ad286d6fed7a580bec36a92c7b7e205322ac407b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To quote Chris Larson:
"""
e.data.getVar() gets a bit old in a large event handler, and it means a
simple handler has to be modified if switching between an event handler
(e.g. RecipeParsed) and anonymous python. I think it would make sense
to restore the 'd' convention here to align with python elsewhere.
It'd just be a convenience, d==e.data, to avoid the common pattern of
setting it at the top of the event handler.
"""
I couldn't find a way to inject 'd' via locals/globals due to the use
of a function parameter so this left __builtins__ as the only way
I could find to make this work.
[YOCTO #7668]
(Bitbake rev: 44ac81e5281fb62ad00e2f79a9d754118ea62526)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than just use d.getVar(X), use the more explict d.getVar(X, False)
since at some point in the future, having the default of expansion would
be nice. This is the first step towards that.
This patch was mostly made using the command:
sed -e 's:\(getVar([^,()]*\)\s*):\1, False):g' -i `grep -ril getVar *`
(Bitbake rev: 659ef95c9b8aced3c4ded81c48bcc0fbde4d429f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The metadata can potentially use such an event to clean up any
"unreachable" data, solving several problems we currently have
where obsolete data may continue to exist in the shared areas.
(Bitbake rev: c5e6f929f3d5eeb7954660dea62611c58b795ff8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we're not going to change the value of FILE, or we know it isn't
going to have changed (ext == bbclass), don't set FILE.
This avoids messy looking history of the variable as well as optimises
parsing speed slightly.
(Bitbake rev: 88e4600aa66dda2e6c807f9d97af8982bcd8817b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake logger now sets a default project if the TOASTER_BRBE
or TOASTER_PROJECT Bitbake variables are not set.
This a necessary step in getting all builds under a project,
as to unify the MANAGED and interactive modes.
Other small fixes are included, related to the size of the
fields in the database.
(Bitbake rev: 5e0bf388f4e5c1cc493ac8264785e631bad2f672)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This refactoring brings the "local_path" of the
layer from the Layer object to the Layer_Version object, which
is more appropriate as different checkouts of the same
Layer may live in different directories.
This enables us to store Recipe file paths relative to a Layer_Version
at all times, aleviating the need to store full file paths in the
database. We also turn the prefix of the path (e.g. virtual:native
path name space) into a pathflag field.
In turn, this solves the problem of mis-identification of tasks based
on the recipe file paths, since we can also match the namespace of the
file paths on the recipe files.
[YOCTO #7594]
(Bitbake rev: ec43dc569e370767c709dec225cbee0c99151c19)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
At least in theory, the order the keys are expanded in can make
a difference, particularly if there is key overlap.
We also want to ensure that any underlying base key is processed
before any overridden version of that variable (FOO before FOO_x)
which helps the update_data removal code I've been testing.
(Bitbake rev: 863b6add24c211d64ba7931647084321f2d65889)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding a dependency when errors occur accessing a file when calling
handle() is not the correct thing to do. THe handle() code calls
resolve_file() which can raise an exception without ever touching
"fn" itself, it has also already marked all the dependencies correctly.
This leads to bogus files being resolved to the local cwd and
hence triggers reparses for no good reason.
The solution is to simply remove the bogus dependency.
(Bitbake rev: 366af3be1cffd64e4a79c15990c1e05869022c14)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst bitbake has done this for a long time, the behaviour of resolving
class files against cwd is not desirable. This can be seen during
base configuration parsing when looking for base.bbclass where a dependency
on cwd is added. If cwd then changes, the cache is invalid and triggers a
re-parse.
The only real option is to drop this entry and if files can't be found, we
fix BBPATH in the cases where it needs fixing. I didn't find any in the
random selection of layers I tested parsing locally.
(Bitbake rev: 508aad9d5db7e51328b1fd6ee53b4bc3720a30b7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the cooker parsing processes each dump an individual profile
which is ok, but means absolute numbers of function calls for a given load
can be tricky to determine as parsing of recipes may go to different pool
threads on different runs.
This change collects up the individual thread parsing results and processes
them into one profile output. The profile processing function in utils
needed tweaks to allow this to work.
(Bitbake rev: d3d2541aacd1ea560da0d8b25a3ea3f0563dee70)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This ensures basic key expansion works and that overlapping
keys generate a log message.
(Bitbake rev: ed5a8954ac923eda9750a636c5bb5b95ffce664f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows us to write tests which ensure a particular action generates
a particular log message.
(Bitbake rev: b30ee0aba51a35a194a4338b988f93ece1ed281c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was incorrect, it *must* be done next to the rename as a previous
may overlap with the current one and we need to detect this case.
I'll add a test case to better catch this problem in future.
(Bitbake rev: 5e9d0911cd86e980ab310cc0d79c9383fbc2c844)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This comparison is interesting even in the case of empty vales. Enabling
this warning actually found a bug in the metadata in avahi. Make the
code handle None specifically and also remove the dead code path in the
second if statement.
(Bitbake rev: a4cd4c56284812efb2a2bc0c8667ddad073f8e94)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By doing this we can take advantage of the expansion cache before
starting write operations on the data store.
(Bitbake rev: 702b42a47904f2378dd819e7463b3206883c2651)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the number of task dependencies change you currently get
a traceback when using diffsigs.
(Bitbake rev: c6798b431571aae18bb8699ac6e3ec75b731d719)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Like the Git fetcher, the Mercurial fetcher shouldn't expect recipes to
provide a checksum. As described [1], recipes using a mercurial
fetcher that don't provide a checksum will fail in a the repository has
previously been downloaded and archived.
Credit to Rafaël Carré for figuring out the bug.
[1] https://www.mail-archive.com/openembedded-devel@lists.openembedded.org/msg41328.html
(Bitbake rev: 2df35a25b4968f64adfa673d5b73442c1a30829d)
Signed-off-by: Ash Charles <ashcharles@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
gitpkgv_revision returns a sortable revision number that can be used
in the PKGV variable for example. To mimic meta-openembedded gitpkgv
behaviour to provide a sortable revision numner, one could set the
following:
PKGV = "1.0+${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"
This would yield a package version like "1.0+69+fb5eb80".
(Bitbake rev: 989c08f62aff7b707c25c692c23284f16506b7bc)
Signed-off-by: Mike Looijmans <mike.looijmans@topic.nl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The idea here is to support package version numbering similar to gitpkgv in
meta-openembedded. This commit is the first step towards such functionality.
The original plan was to add a "get_pretty_srcrev" method to the fetcher, as
per Richard's suggestion [1]. While writing this, I noticed that it would
become a copy of get_srcrev with only two lines changed. So to create something
more Pythonic than a boolean argument and conditionals around the calls to the
fetcher's sortable_revision, I just made the method to be called on the fetcher
an argument to the method. Defaulting to 'sortable_revision' prevents affecting
existing code.
Now if the git fetcher were to implement, say 'gitpkgv_revision' one could
set the following in a recipe:
PKGV="1.2+${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"
and this would yield the same result as gitpkgv's GITPKGV variable.
See for the discussion leading to this change:
[1] http://lists.openembedded.org/pipermail/openembedded-core/2015-January/100345.html
(Bitbake rev: 2f1f4483493cc290f5d2c07f9906e90eaea2f4c1)
Signed-off-by: Mike Looijmans <mike.looijmans@topic.nl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Pass proper repository url without arguments after a semicolon.
Executing checkuri on a rule with git repository in SRC_URI does
not report errors when working offline because wrong repository
url is passed to the ls-remote command. For example
"bitbake -c checkuri glibc" command executes:
"git -c core.fsyncobjectfiles=0 ls-remote git://sourceware.org/git/glibc.git;branch=release/2.21/master"
command in a shell subprocess to determine if url is valid.
Shell subprocess executes in fact 2 commands:
"git -c core.fsyncobjectfiles=0 ls-remote git://sourceware.org/git/glibc.git"
and
"branch=release/2.21/master"
First one returns 127 or 128 depending on error but second one
returns 0 because it is just env variable setup. Therefore we're not catching
connection error.
[YOCTO #7558]
(Bitbake rev: efa44d04137977f883db4a643b0f774e91514722)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix several bugs and add some useful enhancements to make this into a
more generic metadata editing function:
* Support modifying function values (name must be specified ending with
"()")
* Support dropping values by returning None as the new value
* Split out edit_metadata() function to provide same functionality
on a list/iterable
* Pass operation to callback and allow function to return them
* Pass current output lines to callback so they can be modified
* Fix handling of single-quoted values
* Handle :=, =+, .=, and =. operators
* Support arbitrary indent string
* Support indenting by length of assignment (by specifying -1)
* Fix typo in variablename - intentspc -> indentspc
* Expand function docstring to cover arguments / usage
* Add a parameter to enable matching names with overrides applied
* Add some bitbake-selftest tests
Note that this does change the expected signature of the callback
function. The only known caller is in lib/bb/utils.py itself; I doubt
anyone else has made extensive use of this function yet.
(Bitbake rev: 20059e4d5ab9bf0f32c781ccb208da3c95818018)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Fix unchanged assignments being dropped if other lines changed
* Fix not passing variable name from single-line assignments to the
function
* Fix not trimming the trailing quote from values
(Bitbake rev: 0b0c82f49cf2de887967d305768cbd95314bb171)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In certain contexts it can be useful to find the layer that a file (e.g.
a recipe) appears in.
Implements [YOCTO #7723].
(Bitbake rev: 3bf9c8830c5d5eea5502230d5af84ebd87ad5849)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add some tests of mirrors or mirrors to the fetcher unittests.
(Bitbake rev: e33d82bc10283d533f928836d56a6f0af80ea5c1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
First, when building mirror urls we don't do any fetching so we should never
be calling clean functions.
Currently, if a mirror url fails, we don't process it further to see
if there are any mirrors of the mirror.
We should do this even when the mirror url fails, else we may miss out
on valid/useful mappings, particularly in the case of file:// urls.
(Bitbake rev: b7fd3ec9994f664b17fc86423e6e7afac07e897b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow GIT_SMART_HTTP to be passed through to the fetch command so that
servers that cannot use GIT_SMART_HTTP can be used by the fetcher.
(Bitbake rev: e5c97a85bed0436d48eeaac2e32962cfb5371d2f)
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The URL is not sent when _latest_revision generates and exception.
When performing the sanity checks it is not possible to know the URI that failed.
This add the URL when latest_revision generates an exception.
[YOCTO: #7592]
(Bitbake rev: 9f2115b07a55cb14e4a74dc6fbd3707c28a234d0)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add some sanity checks on the parsing state engine when returning data
so that incomplete functions raise parse errors.
This means a recipe doing:
do_somefunction {
echo 1
VAR = "1"
will now raise a ParseError. To get the right file/line information,
__infunc__ was changed to a list.
[YOCTO #7633]
(Bitbake rev: 6b54a72638f57882d4fd5aab96b2752a09e065af)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fetcher calls os.chdir() in a number of places, which can affect
other tests (since the directory it changes into gets deleted) - let's
just put the current directory back to where it was when we're done.
(This fixes bb.tests.Path.test_unsafe_delete_path failing if it was run
as part of a full bitbake-selftest run, where the fetcher tests get to
run before it.)
(Bitbake rev: b1653855c74f86909c9f329ed6d2b10391c28395)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you're interested in using the checked out repository for development
(e.g. in OE with devtool) then you ideally want the origin remote to
point to the repository it was fetched from, so just set that after
cloning.
(As part of this I did a minor refactor so we have one function to
generate the repository URL, which was already in two places.)
Fixes [YOCTO #7756].
(Bitbake rev: 80ecd1c54d4c748cee3a7ce0d64013a346e7671e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases we need to check specifically for siginfo files, in
some cases we need to check for the actual sstate objects themselves.
Therefore make this a parameter to the function. A fallback to the
previous function style is maintained for now.
(Bitbake rev: 18d3a03e1b07c98b2dce46eb94f30de1a2b4320b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BB_ALLOWED_NETWORKS is a list of hosts that the fetcher will be allowed
to use when BB_NO_NETWORK is not set.
If BB_NO_NETWORK is set, then networking is still disabled.
If BB_ALLOWED_NETWORKS is not set, the behaviour remains the same as
today.
If BB_NO_NETWORK is NOT set, and BB_ALLOWED_NETWORKS is configured, then
only the hosts in the list are usable by the fetcher.
eg:
BB_ALLOWED_NETWORKS="yoctoproject.org git.gnu.org"
The fetcher will be able to download from yoctoproject.org, git.gnu.org,
but not ftp.gnu.org or any other hostname that is not in the list.
There is also limited support for wildcards on the beginning of the
hosts, so BB_ALLOWED_NETWORKS="*.gnu.org" with match git.gnu.org and
ftp.gnu.org as well as foo.git.gnu.org
(Bitbake rev: c7263096ba31ba45daeeb9de90c1cb9ebef24a28)
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use hg clone and hg pull to copy the source into the build
directory rather than taring up the cloned repository and
untarring in the destination.
This allows submodules to be cloned. While here, make the default
behaviour keep the hg scm data to match the behaviour of the git
fetcher.
(Bitbake rev: f002b1ca80cb542a4ed0c06c53c914cd5e076565)
Signed-off-by: Daniel McGregor <daniel.mcgregor@vecima.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Setting BB_ENV_EXTRAWHITE to include GIT_SSL_CAINFO is enough to
get "git ls-remote" to work for recipe parsing but it is not enough
to get the fetcher to clone properly.
This is because the fetcher has its own idea about what variables
should be exported in the environment for some operations.
It is desirable to use alternate CA Bundles for internal testing prior
to using public keys for https, so we should allow the GIT_SSL_CAINFO
to pass through.
(Bitbake rev: 40ff92282bbf32cf644b021bf7cbb1f393dbb856)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Moving the code to close the build and instantiate a fresh
buildinfohelper object to BuildCompleted event, as the
CommandCompleted/Failed/Exit events come in too early.
(Bitbake rev: af63abe88327fd5c1b3d7c00a84d9e408ef23285)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch modifies the toasterui to properly return the exit
code based on the errors found in the toaster itself.
The upload event file API call will not delete event logs for which
toasterui showed an error. This will facilitate debugging.
Minor enhancement in the buildinfohelper to reduce the number
of lookups on unknown layer objects (prevented testing of the patch).
(Bitbake rev: 1ddd6a9e4280a4adf971132ff1fe7ec9b3252905)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Introduced build mtime cache structure. Reset it before the build
to prevent bitbake from crashing when build/tmp/stamps hierarchy
is removed.
[YOCTO: #7562]
(Bitbake rev: f8590547a198a78334debdf14bf40acb50c22ecc)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake program and core versions must match.
Moved __version__ from main.py back to bin/bitbake.
Implemented check for version match in bin/bitbake.
(Bitbake rev: 2fe7d8c574ddf6a30278cff1a5a5c4089dc56d6d)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
tbs
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some error messages were lost because BBMainException was
inherited from bb.BBHandledException. When bb.BBHandledException
is processed error messages are not printed as they suppose to
be printed before raising this exception.
Stopped to inherit BBMainException from bb.BBHandledException.
Handled BBMainException in bin/bitbake and printed error message
to the stderr.
(Bitbake rev: c8e2a40c4e9865ebef9936d23644f2602a5c90f5)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add some very basic safeguard against recursively deleting paths such
as / and /home in the event of bugs or user mistakes.
Addresses [YOCTO #7620].
(Bitbake rev: 56cddeb9e1e4d249f84ccd6ef65db245636e38ea)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Implemented processing of ~ in bblayer's paths if HOME
environment variable is approved.
(Bitbake rev: 3b8a656d3ccb543c32696229184ebf12237ad38e)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the subpath parameter to the git fetcher ends with a trailing '/',
bb.utils.prunedir() will be called on '/'...
Fixes [YOCTO #7620].
(Bitbake rev: 380a3fb372c8b0a53dd7528562e6e7a222dc76ef)
Signed-off-by: Anders Darander <anders@chargestorm.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This ensures that _append, _prepend, overrides, etc are functional when used
on sstate variables (e.g. SSTATE_DIR).
[YOCTO #7564]
(Bitbake rev: 2e683c25b856b431198573f7f352d841587275e6)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
self.appendlist is a dict and as such unordered. This can lead to cases
where appends with different names (e.g. x_%.bbappend vs. x_123.bbappend)
can be reordered in application which in turn reorders the variables
that those bbappend files might touch. Reorderd variables changes the sstate
cache signatures causing real world issues.
To avoid this, use a list for the append files instead.
This patch is conservative and just adds a new data structure alongside
the existing one and uses it to resolve the core issue. Later patches
(post release) can handle some of the wider but less problematic ones
(e.g. issues in bitbake-layers flatten).
[YOCTO #7511]
(Bitbake rev: f980f060cd0d1e7fe5011f3c325c1b254f05eccf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since toasterui acts as the user-facing UI, we need
to run sanity checks in order to let the GUI display proper
warnings and stop the build if something is wrong.
(Bitbake rev: 260dd77fa771ae3b777134f4178d344e96b6f3d6)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch prevents tracebacks and instead logs exceptions
that may happen during event processing.
[YOCTO #7216]
(Bitbake rev: 0412631fb4a15ff42bf5ee46a77920fa558ae358)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Moved most of functionality of bin/bitbake to lib/bb/main.py
to be able to call bitbake from python code.
(Bitbake rev: d377f7f88d73f4e5d2dffef03d6acee809827ac6)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch fixes the recipe and layer identification by path
when Toaster uses relative paths.
(Bitbake rev: a92bb33a3ceacab2bfee9df1c39a202832866970)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We improve logging and signalling of errors in the
toaster_ui.log to facilitate debugging on remote systems.
(Bitbake rev: 3cd248f99b90367bd41aab81e255fc1912434890)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It used 5 spaces as the indent.
(Bitbake rev: 162d35ed53d34b28b153adf643044e7f105fcff1)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We modify the toasterui to log relative recipe paths
in order to maintain consistency with data fetched from
the layer sources.
(Bitbake rev: 253d69e88fd68729196ad43c15e8733527d76198)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In build mode, instead of creating our own layer objects,
we identify the layer objects that the build system set up.
[YOCTO #7378]
(Bitbake rev: 22962b540ace6868cb357c0fd13f01ffd24449c4)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Every time the bitbake show versions command (bitbake -s) is run it creates
a 100k log file.
The consolelogfile is disabled for show environment and disabling show
versions would make the behaviour match.
(Bitbake rev: dee0ba94e39ea49c1e9261a5e8932356e3bb7c57)
Signed-off-by: Rob Woolley <rob.woolley@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Any exceptions that occur in calls to logging methods are automatically
suppressed, including exceptions due to broken pipes.
However, the knotty summary messages are printed directly to stdout, which
means that any broken pipes will cause an exception traceback in python.
By wrapping the summary section in a try / catch block we can check for
IOError exceptions caused by broken pipes and let them pass.
(Bitbake rev: 146e7e157f97b676858ecff583dd53800d997253)
Signed-off-by: Rob Woolley <rob.woolley@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The recent inotify changes are causing a 100% cpu usage issue in the
idle handlers. To avoid this, we update the idle functions to optionally
report a float value which is the delay before the function needs to be
called again. 1 second is fine for the inotify handler, in reality its
more like 0.1s due to the default idle function sleep.
This reverts performance regressions of 1.5 minutes on a kernel build
and ~5-6 minutes on a image from scratch.
(Bitbake rev: 0e0ba408c2dce14a0fabd3fdf61d8465a031495b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
[YOCTO #5571] -- https://bugzilla.yoctoproject.org/show_bug.cgi?id=5571
The following workflow (whether accidentally or deliberately) would
previously not result in a checksum error, but would be helpful to do
so:
- Write a recipe with correct checksums
- Fetch the sources for this recipe using bitbake
- Change the checksums
Since the bitbake fetcher writes a done stamp after the initial download
and does not verify the checksums again (even if they are changed in the
recipe afterwards), the change of checksums is silently ignored.
Fix this without the overhead of computing the checksums from scratch on
every do_fetch by storing them in pickled format in the done stamp and
verifying that they still match those in the recipe.
(Bitbake rev: fbd4a0d422cf7f43db2f9eab2e47c41246a9031e)
Signed-off-by: Clemens Lang <clemens.lang@bmw-carit.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch makes sure to delete an idle function that
raises an exception for the xmlrpc server.
The counterpart functionality in the process server was
added with:
commit db50630948.
duthor: Richard Purdie <richard.purdie@linuxfoundation.org>
Date: Wed Aug 20 22:31:06 2014 +0000
bitbake: process: Deal with infinite looping of the server
This patch fixes
[YOCTO #7316]
(Bitbake rev: e7c9a6788d969c901fd6394416ac3936e62c4c72)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch fixes a bug where if the build is force stopped,
subsequent clients cannot connect to the server due to
unnecessary limits on setFeature.
Additionally, we make sure that the state is properly reset
even if the BuildCompleted event firing excepts for some reason.
(Bitbake rev: 0b66b05169688aa4ddc4c54d175bb961b2f27fec)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We roll back the client connection if some error happens,
like during setFeatures, as to leave the server accessible
to other clients.
(Bitbake rev: 4e4a2ee2f05f8741b2e09263e328420363975b02)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The inotify facility monitoring changes to the config files
could be overwhelmed by massive changes to the watched files
while server is running.
This patch adds verification the notification watches to the
server idle functions, in addition to the cooker updateCache
command which executes only infrequently, thus preventing
overflowing the notification buffer.
[YOCTO #7316]
(Bitbake rev: 996e663fd5c254292f44eca46f5fdc95af897f98)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some databases, notably the SQLite3 adapter, force autocommits even if
the autocommit is turned off. The behavious is tracked in this bug:
http://bugs.python.org/issue8145#msg109965
Django refuses to work with autocommit off in this case, so we have
to take the same precautions when using manual transaction support.
[YOCTO #7363]
[YOCTO #7365]
(Bitbake rev: 90231ab63a129fa344d461c2911898ea0f07f206)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch fixes a problem where set sstate scene tasks
were not identified, causing cache attempt not being recorded.
[YOCTO #7223]
(Bitbake rev: 8a326a9a5a08981f1b7960e02fdb8a9436db16fb)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch disables autocommit for inserting build data,
effectively updating all build data in a single transaction.
This is a purely performance improvement patch, as the transaction
will always be commited.
Similar manual transaction handling in the layer source update
method. Added feedback messages during update method.
[YOCTO #7140]
(Bitbake rev: 3978c819e797f857235499a4b8ec238134f1c028)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch brings in cooker log saving and proper download links.
* toasterui will now write the cooker log file if running in managed
mode
* the BuildRequest has a new state, REQ_ARCHIVE, indicating that the
build is completed, and the artifacts are ready to be grabbed
* the runbuild test execution commands will gather needed artifacts,
and save them to a storage directory selected during Toaster setup.
* the build dashboard, project builds and all builds pages have
permanent links for the cooker log
[YOCTO #7220]
[YOCTO #7206]
(Bitbake rev: fad80e36c9da663b000cdf2cb3c75440c6431d84)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add set of PN in data because now latest_versionstring use it for
validate version directory searching.
(Bitbake rev: 2e4a03db967ac1459b2764108fc54c4566a7e371)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix sustition for rc, beta and alpha releses from -N to N weight.
(Bitbake rev: 63a9e60a6d80cfd2693ec1a6359785dc19f98e1f)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for scan every version directory using _check_latest_version
makes code more robust because sometimes upstream projects publish
new directories without files, causing don't find version.
To support this new behaviour remove _check_latest_dir and replace for
_check_latest_version_by_dir,
(Bitbake rev: 1a75b3707743c32eec9d2cf566fb6bbea9f73784)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add _check_latest_version_by_dir this function provides support
for scan every directory newer than current dir in order to get
latest_versionstring, example:
http://somedoamin.com/project/v2.1/http://somedoamin.com/project/v3.0/
Change return of _vercmp from True/False to -1/0/1 to provide test
when current directory is equal to newer directory this helps to
scan the same directory to get minor versions, example:
http://somedoamin.com/project/v2.1/project-v2.1.2.tgzhttp://somedoamin.com/project/v2.1/project-v2.1.6.tgz
(Bitbake rev: 5f7c5eb218a221165f59a0f4dd48d2d97f756193)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to reduce code duplication now compile package_regex in
_init_regexes instead of make this decision at _check_latest_version,
(Bitbake rev: e7284e3ad0e7dd91ed59dfbf8450ef62e89c7e54)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for get group only if exist in regex, this enables to use
this function in _check_latestversion regardless if the regex is generic
or specified by REGEX_URI.
(Bitbake rev: 1127af5b8c458929c4685b0326f86870ed09442e)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've had versioned dependency support in LAYERDEPENDS for quite a long
time, but I can say with pretty good certainty that almost nobody has
used it up to now because it was too strict - the specified version had
to exactly match the version in your configuration or you would get an
error; there was no "greater than or equal" option, which is usually
what you will want given that LAYERVERSION does get bumped from time to
time.
However, users mismatching layer branches and then having their builds
fail later on with some incomprehensible error is still a pretty common
problem. We can't simply use the git branch because not everyone is
always on a branch and the branch names don't always match up (and
that's not an issue). To provide a practical means to address branch
mismatching, I have reworked LAYERDEPENDS version specifications to use
the more familiar "dependency (>= version)" syntax as used with package
dependencies, support non-integer versions, and clarified the error
message a little. If we then take care to bump the version on every
breaking change, it is at least possible to have layers depend on these
changes when they update to match; we can now even support a major.minor
scheme to allow retrospectively adding a version limiter to old branches
when a new branch is created and yet still allow the old branch minor
version to be bumped if needed.
Fixes [YOCTO #5991].
(Bitbake rev: 408be9cdf2b1e32e64ea488d8051a546fb54c144)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This scheme is used for versioning recipes that are pre-release (alpha,
beta, etc.) within OpenEmbedded, so add some tests to ensure the
appropriate comparison results still hold true.
(Bitbake rev: 3a9eefe27f29a4593d6298f0427ac5f3e9183377)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We really want an error rather than the version to just be silently
skipped when the operator is missing (e.g. "somepackage (1.0)" was
specified instead of "somepackage (>= 1.0)".)
(Bitbake rev: b6dc946f477adc40d68da16e2f2580cb3b4a10db)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake -g -u depexp <package> segfaults when DISPLAY is not set
properly. Fix it with a proper check.
[YOCTO #7299]
(Bitbake rev: f35e9bd7b59c180fe9a3d9177efb57b92d9cd373)
Signed-off-by: Maxin B. John <maxin.john@enea.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch improves the logging facilities for toaster in order
to help diagnose bugs that happen on user machines.
The logs are stored now under "/tmp/toaster_$$" where $$ is a
PID-based unique identifier. On shutdown, toaster will automatically
erase all logs unless errors are listed in the log file.
On error, Toaster provides suggestions on what to do.
This patch includes a minor fix found as a result of logging
improvements.
(Bitbake rev: 8a8248f7b7e30469f592e2f8adbf6ce21e8685c5)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Given these assignments:
TEST="a b c d"
TEST_remove = "b d"
TEST evaluates to "a c". However, if the _remove override is given as a
variable:
TEST="a b c d"
FOO = "b d"
TEST_remove = "${FOO}
TEST evaluates to "a b c d", because when FOO is expanded it isn't split into a
list.
Solve this by splitting all members of removeactive once they've been expanded.
[ YOCTO #7272 ]
(Bitbake rev: 207013b6dde82f9654f9be996695c8335b95a288)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The _remove operator isn't working correctly when used with a variable that
expands to several items, so add a test case to exercise this path.
(Bitbake rev: cb2a62a5fbffb358528a85b46c1fc6383286cb9d)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These only have one instance created which means your subsequent datastores
can contain echos of previous ones. Obviously this is not the behaviour
we want/expect. It doesn't affect bitbake too badly as we only have one
datastore, it does massively potentially break our selftests though.
Thanks to Tim Amsell for pointing out the now obvious problem!
(Bitbake rev: 9facf3604759b00e8fe99f929353d46f8b8ba5cb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
ToasterUI filters build artifacts by extension in order
to determine if a build artifact is an image or not.
Using IMAGE_FSTYPES for this purpose is not correct as
the varible value holding image extensions is just a coincidence.
So we just look if the filename contains the "rootfs" magic
string, which is a pretty good approximation.
[YOCTO #7213]
(Bitbake rev: b11e8bd626e0212ee72914529c3d92d1dd718674)
Signed-off-by: Alexandru Damian <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The get/set_taskdata functions are now part of the API of the class,
ensure they exist in the base class definition so the noop handler
works.
[YOCTO #7233]
(Bitbake rev: 9b5b1bd7d77e3f5886f6c557d3b750de1f6d6025)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Users expect operators like _remove to work on this variable. We need
to use expanded_data to ensure this happens correctly.
[YOCTO #7135]
(Bitbake rev: cc4c8478fc547ea0ebf827a8d319496b39f25684)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
event_data would be better named expanded_data, then we can widen its scope
to other places in cooker where we need to access an expanded data store.
We certainly don't want multiple expanded data stores.
(Bitbake rev: 1a3c1c9203e1a1452314954f1cfd771e5c1ce89b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was a report that bitbake -e | less would use 100% cpu when it shouldn't
really. The issue appears to be a bogus file descriptor in the select call. We
shouldn't be blocking if there is event data pending to a *reader* from server
context.
[YOCTO #7138]
(Bitbake rev: 8f166e1a0f3574ae7d1e917a8bb403b87bad15bf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add a call that saves any queued events to the build
[YOCTO #7021]
(Bitbake rev: 4f5b19d453da64749affc1c27ec51b013bedc71a)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When built package regex for use in searching upstream version in sites,
the package name need to be escaped to avoid usage of special regex
char.
For example when search for gtk+, '+' need to be escaped.
(Bitbake rev: 1aa1de4b0c5bd34466e04844bbc371933736be59)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Trap the ENOSPC error and translate it into a human readable error
message, which is good for debugging.
(Bitbake rev: 2b084dff6ff0d274fbbf7ab07022507f7249e427)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We currently add crazy numbers of watches on files. The per user limit is 8192
by default and on a system handling multiple builds, this can be an issue.
We don't need to watch all files individually, we can watch the directory containing
the file instead. This gives better resource utilisation and better performance
further reverting some of the performance regression seen with the introduction
of pyinotify.
(Bitbake rev: a2d441237916a99405b800c1a3dc39f860100a8c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the environment changes, we need memory resident bitbake to adapt to those
changes. This adds in functionality to handle this alongside the configuration
option handling code. This means that the common usage:
MACHINE=X bitbake Y
now works with the memory resident server.
(Bitbake rev: 4d1343010da757a0c126bc22475354da44aaf8e3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tweak the comments on get_srcrev() to better describe its function.
(Bitbake rev: b4d40f1ac7b32990c456cce261f99a5a157b5ae5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A bug slipped in the toaster ui that prevented saving of
build configuration despite the data being retrieved
from the server. This patch fixes the shaming mistake.
[YOCTO #7117]
(Bitbake rev: 8118f465b9f87c66b2a741008f69198ac5fea901)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The import statements here are plain bizarre. Remove them, tweaking
some of the function calls to match current practices. I can't find any
reason these old imports are as they are.
(Bitbake rev: 4c2f1fe51a13ddc97e518327714292af46b9e1ab)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Looking at this function I had no idea what oldfn was, I doubt anyone
else would either without looking up what the caller does. "parentfn"
would seem a more appropriate name so rename it.
(Bitbake rev: fc70ed596703a1aa954223b169d4ad51193a6ec1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Benchmarks show that the introduction of pyinotify regressed
performance. This patch ensures we only call the add_watch() function
for new entries, not ones we've already processed which does improve
performance as measured by "time bitbake -p".
This doesn't completely remove the overhead but it does substantially
reduce it.
(Bitbake rev: 493361f35f6cc332d4ea359a2695622c2c91a9c2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We try and add watches for files that don't exist but if they did, would influence
the parser. The parent directory of these files may not exist, in which case we need
to watch any parent that does exist for changes. This change implements that fallback
handling.
(Bitbake rev: 979ddbe4b7340d7cf2f432f6b1eba1c58d55ff42)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Memory resident bitbake has one current flaw, changes in the base configuration
are not noticed by bitbake. The parsing cache is also refreshed on each invocation
of bitbake (although the mtime cache is not cleared so its pointless).
This change adds in pyinotify support and adds two different watchers, one
for the base configuration and one for the parsed recipes.
Changes in the latter will trigger a reparse (and an update of the mtime cache).
The former will trigger a complete reload of the configuration.
Note that this code will also correctly handle creation of new configuration files
since the __depends and __base_depends variables already track these for cache
correctness purposes.
We could be a little more clever about parsing cache invalidation, right now we just
invalidate the whole thing and recheck. For now, its better than what we have and doesn't
seem to perform that badly though.
For education and QA purposes I can document a workflow that illustrates this:
$ source oe-init-build-env-memres
$ time bitbake bash
[base configuration is loaded, recipes are parsed, bash builds]
$ time bitbake bash
[command returns quickly since all caches are valid]
$ touch ../meta/classes/gettext.bbclass
$ time bitbake bash
[reparse is triggered, time is longer than above]
$ echo 'FOO = "1"' >> conf/local.conf
$ time bitbake bash
[reparse is triggered, but with a base configuration reload too]
As far as changes go, I like this one a lot, it makes memory resident bitbake
truly usable and may be the tweak we need to make it the default.
The new pyinotify dependency is covered in the previous commit.
(Bitbake rev: 0557d03c170fba8d7efe82be1b9641d0eb229213)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When urls ends with trailing slash os.path.basename return "" [1]
and built urldata.localpath only with DL_DIR, it causes that
donestamp is built as DL_DIR + '.done' and seems that ssh resource
was already download.
[YOCTO #6448]
[1] https://docs.python.org/2/library/os.path.html#os.path.basename
(Bitbake rev: 47992591349bab2c12741b937096e41085399087)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the code ignores lightweight tags which has caused some user
complaints. We can't put the right search list in place easily since
the results don't come back in a good order, head happens to sort
before tags.
In the end I refactored the function so we get the complete list of
remotes and then we can filter it ourselves in the order we chose,
including checking for light weight tags, preferring the proper ones.
Hopefully this resolves the issues people have been seeing.
[YOCTO #6881]
(Bitbake rev: 07ad307065bb15a48f0015b9e4a643201abdc283)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A dash character is illegal in function names in sh (but not bash). Since
our shell tasks run under sh and the shell parser is sh based, EXPORT_FUNCTIONS
won't work with class names containing a dash.
We can't change sh, we can ensure the user is warned about the problem
straight away though.
[YOCTO #7006]
(Bitbake rev: 86704281b79e524dccccc88cbf996b299b33bae2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if you reference a file url, its checksum is included in the
task hash, however if you change to a different file at a different
location, perhaps taking advantage of the FILESPATH functionality, the
system will not reparse the file in question and change its checksum to
match the new file.
To correctly handle this, the system not only needs to know if the
existing file still exists or not, but also check the existance
of every file it would have looked at when computing the original file.
We already do this in the bitbake parsing code for class inclusion. This
change uses the same technique to log the file list we looked at and
if files in these locations exist when they previously did not, to
invalidate and reparse the file.
Since data stored in the cache is flattened text, we have to use a string
form of the data and split on the ":" character which is ugly, but is
an internal detail we can improve later if a better method is found.
The cache version changes to trigger a reparse since the previous
cache data is now incompatible.
[YOCTO #7019]
(Bitbake rev: 6c0706a28d72c591f1b75b6e3f3b645859387c7e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases for cache purpoes we not only need to know which file
is going to be used but also which paths were considered. Add a
localpaths method which includes the history.
The core which() funciton already supports this, this just extends
the function to preserve the extra data we need. localpath becomes
just a special case of the case with history.
(Bitbake rev: ea5efeac5c1f7986666c979f789786f29fc1619a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the cooker is in an error state, we shouldn't continue to try parsing.
This fixes an issue where an invalid PR server is detected when bitbake
is started and ensures bitbake exits cleanly rather than hanging.
[YOCTO #6934]
(Bitbake rev: 294bb9cad294423d4f8998405ceff58655f12660)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a little crude as the usage basically ignores that you're doing
an append operation, but for a lot of cases it will be sufficient.
(Bitbake rev: 24a28205ab680b6cc645d97b76c9855920608229)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Fix add-layer error message when a layer is already in BBLAYERS
* Ensure we show an error message if we can't find BBLAYERS at all
(Bitbake rev: 1c743fd2103730e27699dd55efc6914d3b0c3702)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Hardcoding a temp directory is bad practice and leads to races between
the tests. There is no longer any good reason for doing this, drop it
and ensure the files get cleaned up correctly.
(Bitbake rev: 10a47b1ec7470c9e8c4ffe0bb35cdf6d1bb2ee2e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you copy the datastore, then delete a key, it should not exist in
d.keys(). This adds a test to cover the recently found data store bug.
(Bitbake rev: 16d5f40ad20fd08bf7a4d0e36200c739b5a9f59e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The versionstring tests hit the network so should only run when
network tests are enabled.
Also remove the print statement which confuses the test output and
add it to the test failure message instead.
(Bitbake rev: 6c046660cfc4fb3792a42aeafff91a13f68a2e89)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you copy the datastore, then delete a variable, it still shows up
in d.keys() when it should not. This patch addresses the issue.
(Bitbake rev: f28ee1bb03cb32d3757fbef67c9fbe143e3dadfa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to fix multithread usage of latest_versionstring moves package_custom_regex_comp
from class to method level because need to be defined by package.
Remove code for build url's with /download suffix because it's deprecated since you can
specify the download directory using package_regex.inc file.
(Bitbake rev: 231cae9f9b552ec6737795c098d1de426b5adcbc)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a BBFILE_PATTERN_IGNORE_EMPTY variable to allow ignoring the fact
that a regex specified in BBFILE_PATTERN for a particular collection
doesn't match any recipes. This will be used in OpenEmbedded in the
workspace layers created by "devtool" which may not always contain any
recipes (which is not cause for warning the user).
(Bitbake rev: 19c74aaa2836a88d666f0032452fac521689ab6b)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Add a generic edit_metadata_file() function to modify variable
assignments in any metadata file (conf, bb, bbappend) using a callback
for flexibility
* Add a specific edit_bblayers_conf() function to modify
conf/bblayers.conf and add and/or remove layers from the BBLAYERS
value within it.
(Bitbake rev: aa03a28b442549dd8ffe92ae4d6390f62202a76a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a nostamp task is depended on by a non-nostamp task, then we want the
signature of that task to change such that it re-executes afterwards.
This is an unusual situation, but we want this to work in OE in
externalsrc.bbclass so that compilation happens every time it is
requested.
(Bitbake rev: 73498afc3d45beede5b8f24a9acd523a1663b793)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Running bitbake inside make results in the exported environment variable
MAKEOVERRIDES="${-*-command-variables-*-}", which the shell chokes on
when trying to expand it. But of course, it probably shouldn't have been
trying to expand it in the first place -- so just escape the dollar
sign.
(Bitbake rev: 18cd0ce6a55c9065c3f1bf223b47d817b5efcd8f)
Signed-off-by: Richard Tollerton <rich.tollerton@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds a "-w/--write-log" option to bitbake
that writes an event log file for the current build.
The name of the file is passed as a parameter to the "-w"
argument. If the parameter is the empty string '', the file
name is generated in the form bitbake_eventlog_DATE.json,
where DATE is the current date and time, with second precision.
The "-w" option can also be supplied as the BBEVENTLOG
environment variable.
We add a script, toater-eventreplay, that reads an event
log file and loads the data into a Toaster database, creating
a build entry.
We modify the toasterui to fix minor issues with reading
events from an event log file.
Performance impact is undetectable under no-task executed builds.
(Bitbake rev: 1befb4a783bb7b7b387d4b5ee08830d9516f1ac2)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We update and add logs throughout the code in order to help
with development. The extra logging is turned off by default,
but it can be enabled by using environment variables.
All logging happens through the Python logging facilities.
The toaster UI will save a log of all incoming events if the
TOASTER_EVENTLOG variable is set.
If TOASTER_SQLDEBUG is set all DB queries will be logged.
If TOASTER_DEVEL is set and the django-fresh module is available,
the module is enabled to allow auto-reload of pages when the
source is changed.
(Bitbake rev: 10c27450601b4d24bbb273bd0e053498807d1060)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie
Joshua Lock
Ed Bartosh
Michael Wood
Robert Yang
Klauer, Daniel
Olof Johansson
Markus Lehtonen
Aníbal Limón
Derek Straka
Brendan Le Foll
Lucas Dutra Nunes
Elliot Smith
Peter Kjellerstedt
Paul Eggleton
brian avery
bavery
Alexander Shashkevich
Mirela Rabulea
Christopher Larson
Schroeder, Henning
Aníbal Limón
Felipe F. Tonello
Phil Blundell
Sujith H
Andre McCurdy
Ross Burton
Kristian Amlie
Chen Qi
Juro Bystricky
Alexander Kanavin
Jens Rehsack
Logan Buchy
Romain Perier
Mark Hatle
Alex Franco
Benjamin Esquivel
Randy Witt
Alexandru DAMIAN
Mariano Lopez
Leonardo Sandoval
Ash Charles
Mike Looijmans
Marcin Smoczyński
Liam R. Howlett
Daniel McGregor
Jason Wessel
Anders Darander
Christopher Larson
Rob Woolley
Clemens Lang
Maxin B. John
Pascal Bach
Andy Voltz
Richard Tollerton