If there is a space in a directory name containing a file in file-checksums
(e.g. from a file:// url), you currently get tracebacks from bitbake. This
improves the code to handle colons and spaces in the file-checksums names
since it possible to figure out the correct names.
[YOCTO #8267]
(Bitbake rev: 87282b283921a58426f24fb21151db457c5bca66)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a new bb.utils.signal_on_parent_exit() function so that a process
can register to recieve a signal when the parent dies. There is no
POSIX standard for this and the implementation is Linux specific.
Alternatives would be having an open pipe or polling os.getppid()
for changes but this seems more effective and less invasive to most
of bitbake's code structure.
We need to be able to determine when parents die to ensure child
processes stop running in a variety of circumstances to avoid
locks being held and ensure clean shutdown.
Roughly based on https://gist.github.com/evansd/2346614
(Bitbake rev: 34974f5e30e9b09c016481e4c81c156a5f379784)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if you send a SIGTERM to the bitbake UI process, the system basically
hangs if tasks are executing. This is because the server process doesn't
actually try any kind of shutdown before exiting.
This patch trys executing a stateForceShutdown command first, which is
enough to stop any active tasks before the system exits.
I also noticed that terminate can execute multiple times, once at SIGTERM
from the handler and once from the real exit. Double execution leads to
stack traces and potential hangs (writes to dead pipes), so ensure
the code only can run once.
With these fixes, bitbake much more correctly deals with SIGTERM to the
UI process.
(Bitbake rev: 1032ddddbe3241da02ebb3608a1c40f9123b9e80)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When our clone exists, but is out of date, and the attempt to update it fails,
we don't necessarily want to remove the entire clone, particularly if it's
a large repository.
(Bitbake rev: 19af272ba5256653edeff6acbceeb09e3e478d61)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the main fetch method doesn't support checksums, the user will not be
defining them in the recipe, so we don't want to check them for
premirrors/mirrors either. This ensures that we never error due to missing
checksums on a git mirror tarball.
(Bitbake rev: 24c79bbed361b37f12d3351af13602e3d4386f4c)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using an absolute file URI, there's no host, and the path starts with
'/', the dir under ${DL_DIR}/git2/ ends up starting with '.', so is hidden.
Remove any leading '.' to fix this.
(Bitbake rev: 8dce6964d56b36a77fb113f2ad496cc992a5ff36)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Show the user only the portion of the traceback which was from the metadata,
nothing from bitbake's internal calls.
(Bitbake rev: c45054aef03393fa0bf70e853ddcfc55988493cf)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This gives us the needed context of the original ExpansionError, which is
invaluable when we have a chain of function calls in the expansion.
(Bitbake rev: c514b6fbea77ede1b7871b89592a33ed39b1d71c)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The output of "bitbake, -c listtasks pkg" lists tasks with their real names
(starting with "do_"), but then "bitbake -c do_task" fails, as "do_" always
gets unconditionally prepended to task names. This patch handles this error
by checking whether a task starts with "do_" prior to prepending it with it
when the task runlist is being constructed (and a few other corner cases).
[YOCTO #7818]
(Bitbake rev: dd3050ceef37ac556546e940aa596ce96ef6c8df)
Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you start and suspend a bitbake execution so the bitbake lock is held,
then try and run "bitbake -w '' X", you will see bitbake return an error exit
code but print no message about what happened at all.
The reason is that the -w option creates a "UI" which swallows the messages. The
code which handles this exit failure mode thinks a UI has printed the messages
and therefore doesn't do so.
This adds in an extra parameter to the UI registration code so that we
can figure out whether its a primary UI or not and base decisions on whether
to display information on that instead. This fixes the error shown above and
some bizarre failures on the Yocto Project Autobuilder.
[YOCTO #8239]
(Bitbake rev: d1d60a68c2de40c2984d5040d14251c1be121b0b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake utils' movefile is now prone to malform the destination
file with duplicated file name strings. Fixing it to force a file
name append iff the dest argument is a dir not a file name
(Bitbake rev: 38dd27f7191da002a16c561be3790ce487045b01)
Signed-off-by: Benjamin Esquivel <benjamin.esquivel@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Command line builds are associated with a "default project"
(as we currently require a build to have a project). This
acts as a container for builds initiated outside Toaster.
Currently, this project is marked as the default by its ID
being 0. However, this doesn't work with MySQL, as MySQL
won't allow 0 in a foreign key which references an
autoincrement field.
Instead, use an is_default field to track the default Project
for builds initiated outside Toaster.
Add a method to fetch this default project, rather than fetching
a project with a magic ID.
Add this default project in a migration, rather than as a side
effect of a get_or_create() style method.
Also ensure that builds always have a project explicitly assigned
to avoid any magic with a build's project foreign key defaulting to
0 (as it no longer does).
[YOCTO #7932]
(Bitbake rev: 71b709a1bbc26d89d61873763b467d21e625b274)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When moving a file via the os.rename function, it was missing the
destination file name which caused an OSError
[YOCTO#8180]
(Bitbake rev: b147ba0341d87e077bd2b09ef4355976ecd2d26b)
Signed-off-by: Benjamin Esquivel <benjamin.esquivel@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We want addtask to be able to bring back a deleted task, but we don't want its
previous dependencies to come back with it, so rather than marking a task as
deleted and then skipping tasks marked as such, actually delete the task and
its dependency information in deltask.
While we're in that part of the code, also fix a couple 'not foo in bar'
instances to 'foo not in bar', which is preferred in python.
(Bitbake rev: 94b3f3d6bdfbfa47f7eb3c3de64940a145b2ddd1)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow any listeners for this event (such as buildhistory.bbclass in
OpenEmbedded) to find out if the build was interrupted rather than
completing normally. The value will be 0 if not interrupted, 1 if
interrupted waiting for remaining tasks to complete, or 2 if force
interrupted (stopping any running tasks immediately).
(Bitbake rev: df2b778efd2ecc48f6c5a3ed446f6459f2250035)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're importing the server and UI modules in order to check they are
valid, but it turns out that that has some nasty side-effects. We don't
actually need to do this except when --help is passed or the module
doesn't exist, so rearrange the code so that we only do the module
listing in those two cases.
Additionally, let's just go ahead and catch all errors on import; we
really don't care to have them cause a failure now.
(Bitbake rev: c9dc6d9c86e8b887821a6d00346bd0b09e1da97c)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This exception was triggered in toaster from recent changes and is completely
breaking the whole of bitbake. Add the exception to the list so at least only
toaster is affected.
(Bitbake rev: f64def7cb6069dc1134fcd546bb59e4030c7376f)
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch fixes pylint issues in the toaster build data logger,
toasterui. The following types of warnings are touched here:
* fixing imports
* unused variables are set to _
* logger calls now use lazy evaluation instead of formatting
the string
* correct whitespace identation
* removes unneeded "pass" statements, and extra parantheses
* disable specific pylint warnings when decideing to override
them
(Bitbake rev: 947d47f15048baa967f88e03d80014e88ce152aa)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prompted by issues discovered during running pylint on
the toaster sources, this patch fixes:
* missing import in toaster ui
* improper call of function in toaster ui (logger.debug)
* improper function definitions in bbcontroller
* invalid references to objects in bldcontrol.models
* proper initialization of object fields in ToasterTables
Also inhibiting specific pylint errors in files where
the problems are mis-identified.
(Bitbake rev: 1c71955c416fb68455f7f70669aba4202c411807)
Signed-off-by: Alexandru Damian <alexandru.damian@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake-layers needs to map recipe and class files to the layer they
came from within the show-recipes and show-overlayed commands. However,
it turns out that mapping a file to the layer it came from is not as
trivial as it might seem. To do it properly we need to match the path to
an entry in BBFILES then map that to the collection name using
BBFILE_PATTERN, then map that to the actual layer using variable
history. If it doesn't match any entry in BBFILES, then we can fall back
to BBFILE_PATTERN (to handle classes and conf files).
This fixes the layer name not showing up properly in the output of the
show-recipes and show-overlayed commands for recipes in layers such as
meta-intel that have subdirectories in BBFILE_PATTERN. It also fixes the
priority not showing up in show-layers for such layers.
As part of this I've added a function to VariableHistory which for a
space-separated list variable gives you a dict mapping the items added
to the files in which they were added. I've also fixed
bb.utils.get_file_layer() and reduced some of the duplication by using
this function in bitbake-layers. Also fixes the priority not showing up
for layers such as meta-intel
Fixes [YOCTO #8160].
(Bitbake rev: e852f6cabd7489585477ab567a1afeb2252377ac)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Provide us with a means of showing the list of UIs / server choices for
the command line help, and do the processing in one place for both.
(Bitbake rev: 24035c1daad5a904c3372d21d44191ee8182338f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a file no longer exists, drop it from the cache silently instead of
generating a traceback. This was visible in some cases when a recipe was
deleted when bitbake was resident in memory.
(Bitbake rev: fe105b9042bdac4afd9f38fcf92bfdc2c04ec23f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now we have the bbappends list and all users have been converted over to
use it, we don't need this anymore.
(Bitbake rev: 279770c42d4c63aa2cebce331b55a92a564b50ac)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst collecting this list on the fly may be quicker, doing so within a
function that's meant to *query* the list of bbappends is poor practice.
(Bitbake rev: 5c12aa3b0010d7d1733e54a0ca7d0af465454210)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we can't connect to the server we should error out, because it might
not be that the server is actually dead - it might just be unable to
execute commands.
(Bitbake rev: d4b921676859d6ba4e1922fa4682ee941652f483)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The prefile/postfile options weren't working in memory resident mode
because they weren't being passed through to the server, so ensure that
they do get passed through and that the server is reset when the values
come through.
(Bitbake rev: a3f7dc042fc7b1c308bfd248431930eb8ba50326)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a means of ensuring that synchronous commands that read the results
of the configuration trigger a reparse of the configuration if any
underlying files have changed.
(Bitbake rev: aaf3cc024315450c1674819edf2a4e5cdf293f35)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Unfortunately we were acting on inotify notifications about any files
changing within the watched directories, not just the ones we actually
care about. In OE the build directory is in BBPATH and hence it gets
watched, and we write things like bitbake.lock and
bitbake-cookerdaemon.log to that directory, hence effectively
notifications were being tripped on every bitbake invocation. To avoid
this, record which file/subdirectory we're interested in against each
watched directory so we can ignore any events for files/subdirectories
we don't care about.
Additionally, if we move up to the parent dir, ensure we haven't already
seen it before adding a watch on it (we were previously calling
watcher.add_watch() on the same directory multiple times before in a
typical OE configuration).
(Bitbake rev: bc39b8da34c046b629c43fd0a8cac2efbf1c060f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If sys.exit() is called within a command run over XMLRPC, the XMLRPC
server is effectively trashed (apparently listening but no longer able
to respond to commands). We need to intercept the SystemExit exception
and deal with it as we would any other exception.
(Bitbake rev: 95e391acbc3b4efd6c77637a1ce815012ae0f09b)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With python you should not assign a list as the default value of a
function parameter - because a list is mutable, the result will be that
the first time a value is passed it will actually modify the default.
Reference:
http://docs.python-guide.org/en/latest/writing/gotchas/#mutable-default-arguments
(Bitbake rev: 7859f7388f2e3f675d0e1527cfde18625f36f637)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently bitbake is the only one that knows the relation
between PN and PROVIDES. In some cases it is needed to have
this relation in the data store (the bootloader it's a good
case).
This adds the PROVIDES to the taskdata, so it would be in
the data store as one field of BB_TASKDEPDATA.
(Bitbake rev: a660787311d2961c66c0443bf0e2e094c9baef1b)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you load memory resident bitbake, then change bblayers.conf or other
configuration files, then execute your first command, bitbake will not
notice the changes. This is because it adds the file watches during
inital parsing, which happens at the time the first command is run.
To fix this, we move the addition of the file watches to earlier in
the process, so bitbake then does track them correctly. This also
avoids some issues Paul was seeing with tinfoil2.
(Bitbake rev: b9375c73e736003e66575969c9ea244403e47aeb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've had a long-standing bug where a legitimate error reading a file (IOError
or OSError) is always suppressed as though it was a 'file not found' case. As
a concrete example, if you do a `chmod 000 conf/local.conf`, it'll silently
not parse local.conf, rather than erroring to let the user know about the
problem.
Fix this by handling the ENOENT case specifically.
(Bitbake rev: e691312a3add222b04e7b2f52f8df6abcb9068bf)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tracebacks are of extremely limited usefulness in this context. The exceptions
carry the necessary context already, and the user doesn't care about the calls
in bitbake internals that led to an expansion or parse failure.
(Bitbake rev: 9b95fa94eaae452ac7814f1e67c2f7a6314c52f1)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Set logging level for the Toaster logger to get messages
to toaster_ui.log
Reverted previous workaround.
[YOCTO: #7965]
(Bitbake rev: f1d8e0ff7233f17cff3a56db10d2443b392b407d)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
[\.-_] means "any character between . and _"
What was meant here is certainly "any character from the three characters .-_"
(Bitbake rev: af13eaba627f199f91c048c435b9dbe19c79527f)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the PR server or indeed any other child process takes some time to
exit (which it sometimes does when saving its database), it can end up
holding bitbake.lock after the UI exits, which led to errors if you ran
bitbake commands successively - we saw this when running the PR server
oe-selftest tests in OE-Core. The recent attempt to fix this wasn't
quite right and ended up breaking memory resident bitbake. This time we
close the lock file when cooker shuts down (inside the UI process)
instead of unlocking it, and this is done in the cooker code rather than
the actual UI code so it doesn't matter which UI is in use. Additionally
we report that we're waiting for the lock to be released, using lsof or
fuser if available to list the processes with the lock open.
The 'magic' in the locking is due to all spawned subprocesses of bitbake
holding an open file descriptor to the bitbake.lock. It is automatically
unlocked when all those fds close the file (as all the processes terminate).
We close the UI copy of the lock explicitly, then close the server process
copy, any remaining open copy is therefore some proess exiting.
(The reproducer for the problem is to set PRSERV_HOST = "localhost:0"
and add a call to time.sleep(20) after self.server_close() in
lib/prserv/serv.py, then run "bitbake -p; bitbake -p" ).
Cleanup work done by Paul Eggleton <paul.eggleton@linux.intel.com>.
This reverts bitbake commit 69ecd15aece54753154950c55d7af42f85ad8606 and
e97a9f1528d77503b5c93e48e3de9933fbb9f3cd.
(Bitbake rev: a29780bd43f74b7326fe788dbd65177b86806fcf)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since shell scripts run with 'set -e' the final exit at the end of the
script can only be returning 0. However, for correctness and to follow
the original intention of the 'cleanup' commands, let's fix the typo
and return the success of the last shell function rather than the
success of unhooking the exit trap handler.
(Bitbake rev: bef724057f1ea81571dd3ac5a9cf862f818434b5)
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Calling record() for each override alteration is slow. Since we now expand
overrides dynamically we don't have to record the log data at each alteration,
we can instead print it directly from the existing data stores at variable
history print time using the exact same data stores.
This massively improves performance of the data store when parsing
with bitbake -e for example, it will improve memory overhead as well.
The only downside is that VariableHistory has to poke into the datastore
for some of its data but that seems an acceptable tradeoff rather than
double caching.
(Bitbake rev: 100b447a161ef20fa559e39516cd32fa78e38262)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As things stand now, bitbake -e (which turns on all the caller tracking)
of OE-Core generates around 9.5 million stat calls which is slow and the
largest single thing on the profile data.
This is because infer_caller_details() calls traceback.extract_stack()
which adds line contents to the traceback. This in turn calls python's
internal linecache code which calls stat on every file for every callback.
We don't even use that info. We only even want a single frame of the stack.
Instead, open code for the pieces of information we need. Also, only
obtain the stack once for both halves of the infer_caller_details()
code.
This reduces the number of stat calls to around 0.5 million and significantly
improves parsing with bitbake -e.
(Bitbake rev: 7be76d8f79ea92fd4bd36146eb9a4b86551b526d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We should use the d.xxxVar syntax rather than the older function style.
Also replace 0/1 with the more pythonic True/False.
No functionality changes.
(Bitbake rev: 90fdd69cca951f8bd2ff634f3b42fccd4fc03095)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Minor formatting improvement in generated shell scripts.
Add a newline after the last export emitted by emit_func. Also, remove
chance of putting an extra newline in the middle of the exports, since
we never want to do that.
(Bitbake rev: 5117d9b5d32d5d81adf70fa3e3feac9cef654240)
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
One drawback to tinfoil is that once a cooker is created, it will hold
the cooker lock and stop any other bitbake execution against a directory.
Add a shutdown method to tinfoil, allowing other users to use
the build directory after the tinfoil usage is no longer needed.
(Bitbake rev: e44ce85fe551677fc0dcc1da4f789a0c13093ff1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
One data store changing a variable poked through into a different data
store. This test case replicates that issue where the value 'B' would
become unset/disappear.
We also enhance parsehelper to generate files with an optional suffix
such as bbclass.
(Bitbake rev: 5c4179f58a4e04f1c354df5f17d1860eb403f0ac)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using deepcopy() caused a major performance regression. Turns
out we can work with a shallow copy as long as we force the
recreation of any list that we change, effectively a poor mans
CoW. This isn't too invasive and avoids the huge overhead of
deepcopy so this seems like the best way to have performance and
a working data store.
(Bitbake rev: 32dff4749c32f32e947c42c86f8357b8b607354b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Change the test name to be more specific on what is tested.
(Bitbake rev: 3e39156bc330c4c726058a5b9c13d33e2daad89f)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now latest_versionstring method returns (version, revision) for comply
the new return convention needed by SCM's like git get the current
revision.
bb/tests/fetch.py: Updated wget latest_versionstring test for comply new
convention.
[YOCTO #7605]
(Bitbake rev: 8d454646cbe1b04758ca178d8c6fcfd02b818b7b)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to know the revision associated with the upstream version in
SCM like git, this change broke return convention, oe-core recipeutils
get_recipe_upstream_version need to be updated.
tests/fetch.py: Updated git latest_versionstring test for comply new
convention.
[YOCTO #7605]
(Bitbake rev: fd175dc90024c503134c11cbd83e77d88c406ac8)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Don't limit the tag search for only tags end with ^{}.
(Bitbake rev: 7006ab313766344cf33481228465082ed5977d28)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
dbus sets RDEPENDS_dbus-ptest_class-target = "X". nativesdk as
BBCLASSEXTEND was expanding and setting various RDEPENDS variables
which were clearing overrides queued against RDEPENDS. The problem
was that this was leaking into other data contexts such as the
target case. This was because overridedata was a shallow copy.
Replacing the shallow copy with a deep copy avoids this problem of
leakage between the data stores at a small performance penalty.
(Bitbake rev: 83e2923cd3c8f2e25987132f85b06ce72bc941d7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The recent change to connect through the shell logging functions had an
unexpected side-effect - bb.error() and bb.fatal() cause a flag to be
set internally such that BitBake's UI will not print the full task log
on failure; unfortunately we have in places within the OpenEmbedded
metadata called these shell logging functions under error situations
where we still want to see the full log (i.e., the message we're sending
doesn't include the full error). Thus, provide a mechanism to fatally
exit with an error but unset the flag, utilising the built-in python
logging functionality that allows extra values to be passed in the log
record.
(Bitbake rev: e561b997c55e8537d82aa1339adfff4505cc38b7)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're calling select() to find the fds that have data available, so we
really ought to check the return to see if the extra file is in there
before trying to read from it. This is part of the fix for the
performance regression that this code introduced (5-10 minutes extra in
a reasonable size OE build); the rest is down to an issue in the way
that pseudo currently handles FIFOs and will need to be addressed there,
see: https://bugzilla.yoctoproject.org/show_bug.cgi?id=7993
Solution suggested by Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 3e2db8d7eaa0f14813213d1c95d3ee9efd97dc9d)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a variety of tests which were found to be useful when working
on the data store recently.
(Bitbake rev: 5c5f8da509f6bbc1fad263462142519ef3d54a35)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that overrides get expanded 'on the fly', change appendVar
and prependVar to work using _append and _prepend, else we'd have
to re-implement pieces of getVar and the timing of expansions
becomes problematic.
Using _append/_prepend equivalence gives the behaviour users likley
expect from these functions.
(Bitbake rev: 40d661aaf7a563c6447b073310c5f2fdae6ca3d0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The expand cache can be cleared in the middle of getVar now
due to the use of operations like delVar. We therefore need
to check if variables are in the cache before accessing in
case it was cleared.
(Bitbake rev: a97ce216cfe44136f742383542954bfce027831e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
d.keys() was not returning items which had no base content but the
variable came into existance through overrides. We have to process
self.overridedata to find these other variables.
(Bitbake rev: 77fc57c610d7f65f7e328ad555f525b2f8bdf1d7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This performs better than continually regeneratiing the set().
Also only use set comparisions when its necessary to save some overhead
as issubset() is slower in the single item case.
(Bitbake rev: db9a4eedcf78969ec50bf0e4a8defba8ff0daa4a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When expanding OVERRIDES, its possible someone might try and override a variable
that is used in OVERRIDES. This could lead to infinite recursion. Add in
guards against this.
(Bitbake rev: 07d773369f571028c2cf82dd1f65d9731af6d00e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Updating the value of OVERRIDES whenever it changes turns out to be
extremely expensve/pointless. Instead, clear its value and re-establish
the value when we're going to use it.
This gives significant speed back.
(Bitbake rev: 41cf8d0c92d2d8a33fdad0921e424a0024914be1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using an internal flag for override mapping turns out to be slower
than is optimal, not least as we don't want the keys list to list
variables that have no value other than a potential override expansion.
Maintinaing a seperate cache is therefore more optimal from a speed
perspective.
(Bitbake rev: 1a39ccf66b30b54e1274669cb0e1dc8fc72dba49)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Trying to look up a variable called 'copy' in COW is problematic due
to internal implmentation details, at least avoid tracebacks from this.
Also don't duplicate override history (which is an atrefact of changed
override behaviour) as otherwise the bitbake -e output is convoluted.
(Bitbake rev: dddff5b7b8e6c18515b43389cef35503468b843d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When parsing we find problems if we clear prepends/appends when
setting variables during the initial parsing phases. Later, we actively
want to do this (in what would be post finalisation previously).
To handle this, pass a parsing flag to the operations to control
the correct behaviour for the context.
(Bitbake rev: ae87f5b8bf16191b3201cfb445062938eab992a0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Compeltely remove the replaces functionality and move all overrides
handling to getVar time. We can move the cookie cache into a hidden
flag within the variables themselves.
This removes the need for any of the internal_finalize steps.
This obsolete the need for the _seen_overrides COW cache.
(Bitbake rev: 2a0b73110bede91777ada54d1d89b45fb6034b6c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than repeatedly expanding the value of OVERRIDES, cache
the value and update it when things change.
There were also some bugs introduced in the replaces functionality
which this approach fixes by ensuring the replaces data is updated
at the correct points.
(Bitbake rev: f3b7c3e054ce230ea5c2db5813390383e8dfd6db)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These write operations should clear the expand cache since they can
influence returned variable values but currently do not. Fix this.
(Bitbake rev: a075332c9e13be35f1ae84adc6b29e9137a487ff)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than the heavy lifting in internal_finalize, move the bulk of
the functionality to getVar and rely on a new internal replaces variable
to keep track of any mappings that are needed. This removes the need
for the COW _special_values cache.
This change in functionality also implies we need to track any changes
not only to OVERRIDES but also any variable which OVERIDES depends upon.
Add code to handle this.
Explicitly list FILE as a value OVERRIDES depends upon since it doesn't
automatically get detected and is of key importance to OVERRIDES,
otherwise PN doesn't update when FILE changes.
(Bitbake rev: a6f1377ce3386d274882072d1ae6da3b1834149b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move the update_data functionality into internal data store operations
so the main finalize (update_data) call is a nop.
To make this work we need to call the internal finalization function
whenever OVERRIDES is changed to ensure values get updated correctly.
This has performance issues but the subsequant patches look into this.
(Bitbake rev: 546d9932d6c4413824319a9d780c0d77d2725f4a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since Python 2.7.9 ssl cert validation is enabled by default
see PEP-0476, this causes verification errors on some https
servers so disable by default.
(Bitbake rev: e177170200ece76b36e3f7d5597651fdef67736f)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Only use ProxyHandler opener when exists proxies in env.
(Bitbake rev: 0f062c5d99e12ce20c0e46a8e602448032144ff6)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2/__init__.py: Add connection_cache param in Fetch __init__.
In order to pass connection cache object to checkstatus method.
[YOCTO #7796]
(Bitbake rev: 9fa6407e6cefe66c77467419a8040d6957a6bb01)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Set BatchMode=yes instead of PasswordAuthentication=no. This will make
sftp fail immediately, not only when SSH requires interactive
authentication, but also on errors related to host key verifcation.
(Bitbake rev: 31305853a177735cc9c4553ea8905cd0acfcb100)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Create a branch and named as upstream branch when checkout source
* Set the branch to track remote branch.
(Bitbake rev: 1ba20e4fe9c884515b200589fe379ad5eeda10bd)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you have a situation where you are getting a "Nothing PROVIDES" error
(for example when something you request to build DEPENDS on something
that has been skipped or doesn't exist) it would be useful to be able to
use bitbake -g or bitbake -e to debug it, but currently both of those
are blocked by the error.
This patch adds an "allowincomplete" option to taskdata and uses this
for the -e/-g bitbake options. The NoProvider errors are still
printed and bitbake does return an error exist code but the environment
and task graph files are generated.
[YOCTO #7623]
(Bitbake rev: 2ca36a9f088438a8d1db44119c704f9480b04298)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create a fifo under ${T} for each running task that can be used by the
task to send events back to BitBake. The purpose of this is to allow
OpenEmbedded's logging.bbclass (which provides shell function
equivalents for bb.warn(), bb.note() etc.) to have those functions
trigger the appropriate events within BitBake so that messages are shown
through the BitBake UI rather than just in the task log; to do that we
just call the python functions.
Part of the fix for [YOCTO #5275].
(Bitbake rev: bae330a1f8a59a912eca71177b3c6ba7c92a2f38)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to pass connection cache object to checkstatus function
add fetch parameter.
(Bitbake rev: fbb9c6f5538084e125b58118a86968908e6f895b)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
FetchConnectionCache class acts as a container for socket connections
useful when implement connection cache into fetcher modules.
(Bitbake rev: 454da2cd17539ceb9caad6d76f034757e44ee12f)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before exiting the UI, unlocks the bitbake.lock owned by cooker; this
way consecutive bitbake executions can lock it again without trouble.
[Yocto #7941]
(Bitbake rev: 69ecd15aece54753154950c55d7af42f85ad8606)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Function which calls cooker's unlock method, which in turn unlocks bitbake.lock
file.
(Bitbake rev: e97a9f1528d77503b5c93e48e3de9933fbb9f3cd)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, once set, DATE and TIME in the metadata remain unchanged.
This is suboptimal for cases where cooker is reused. This ties setting
the variables into the BuildStarted event which seems like a more
appropriate time to do so.
It also changes BUILDNAME to be based off DATE/TIME by default if not
already set so that the data is more consistent. We therefore need to
expand the value rather than the previous default of not doing so.
This change does mean the date/time values are in sync across all
variables too.
It does mean bitbake now has special knowledge of DATE/TIME but that
would seen unavoidable (other than doing this in event handlers which
has its own set of downsides).
[YOCTO #5187]
(Bitbake rev: f883cf240266ee7be2cbd8971a8164cf4df9e372)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The incredibly useful recipe:do_task syntax on the command line isn't documented
at all. This isn't much but it's better than nothing.
(Bitbake rev: 7f4c07886ecff4ac77fdd2165bedd179099fcf19)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is nessary when specified branch with submodules is different then
default (master) branch.
[YOCTO #7771]
(Bitbake rev: f7b0b5e33e00f3ce0744322eee93835ee76bf184)
Signed-off-by: Felipe F. Tonello <eu@felipetonello.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make sure the error value is initialised and update the string match
that we have to identify bitbake's current startup status.
Patch contribution from Eduard Bartosh
(Bitbake rev: 915ba08a8a3013e9787e564f2ffd8698c948f433)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These aid when debugging parts of the codeparser cache since the
object contents becomes identifiable.
(Bitbake rev: 344b098c7eafc2bcc5c6b44ea47985bc0cb446b5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The main parser and other code copes with empty python functions but
the python codeparser does not. Fix this to avoid errors with code like:
python do_build () {
}
which currently results in the obtuse:
"Failure expanding variable do_build: IndexError: string index out of range"
[YOCTO #7829]
(Bitbake rev: e4f594c670189e04d58ce7d160fc1d86123620af)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
* do_unpack error:
abort: repository DL_DIR/hg/vim.googlecode.com/hg/vim not found!
* The mirror tarball doesn't work
- Add the build_mirror_data to create the tarball
- Unpack the mirror tarball when needed
* The hg files will put in the dir like git: DL_DIR/hg, it was
DL_DIR/hg/path/to/src/uri/path in the past.
(Bitbake rev: 378647281ae883dd726f1e1b775dd35ef6a0e8d1)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fetching directories with p4 worked but single files did not. This
patch from Helmut Auer (helmut.auer@harman.com) fixes that issue.
[YOCTO #7891]
(Bitbake rev: 39da6579901c62a83ed9319c2016c58fbbc108fe)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Events can call each other recursively, e.g. an event handler can call
bb.note which in turn generates another event. If these loop, it
can lead to multiple deletions of 'd' from __builtins__ which
can fail since __builtins__ is global scope.
Add handling to only remove 'd' when we added it and it wasn't already
present.
(Bitbake rev: b45952650ce8f470f124df36185b79e0d3a1783a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some handlers hook on BuildComplete so it avoids certain event races
to finish the command after the BuildComplete event is sent out.
This means the UI is available to handle events until the command
completes.
What appears to be a race on one of the sanity tests for event handlers
triggered this change although the failure is hard to reproduce.
[YOCTO #7921]
(Bitbake rev: e42d7c47a06fbb5981e0313478c8e3656b99f4e7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch verifies that BRBE is set before trying to use
it to read the checkout paths. This is needed for builds
ran outside Toaster control.
(Bitbake rev: e04807cd3135c9de96cc7f79245f329c24618b85)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We remove the "timespent", "errors_no" and "warnings_no" fields
in favor of computing the needed values at runtime. This prevents
inconsistencies in the UI.
Also removeing all references to BuildRequests from the interface -
all build details now display in the build dashboard.
Minor fixes related to data logging.
(Bitbake rev: 44f37394ed3e4ca02f940be172fe4395b0ee0f7d)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a set of fixes that repair the interface after
we switched from displaying BuildRequest data to Build data
in the formerly "managed" mode.
(Bitbake rev: 57f790b0c56297af8c83d5def8461bd5d61fe4af)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds logic to complete changing the interface
from showing BuildRequests to showing Build data.
The BuildRequest data is now transformed in Build data with
proper Toaster exceptions being recorded instead of listing
problems during startup as build errors.
(Bitbake rev: 51a41172d0b390370f9a38696b1ac65666ada4d2)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake does really weird things with negative or zero numbers of threads
which is confusing to the user. Add a sanity check for this.
When you have code doing arithmetic on the values and a VM reconfigures
to only a single thread, negative numbers are easier than you'd think.
(Bitbake rev: 32166ac3c85ff3c04081580ae76bd63590d6ff3e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we see a BBHandledException in runqueue, the understanding is the system
handled it, printing a log and traceback is just confusing.
Therefore only print these in the cases where its an unknown/unhandled
exception.
(Bitbake rev: 29d28e22ce431c3d3aabdb88ff4d8cca67a1cfad)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we see a BBHandledException in the idle handler, the understanding
is the system handled it, printing a log and traceback is just confusing.
Therefore only print these in the cases where its an unknown/unhandled
exception.
(Bitbake rev: d88ecc2bc44dce8fd92ca3a2c0fd4124a5e464fa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if the fakeworker failes to start the output from bitbake is confusing.
Improve the error handling to give a clear indication of what failed.
Patch from Chris Larson.
(Bitbake rev: ad286d6fed7a580bec36a92c7b7e205322ac407b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To quote Chris Larson:
"""
e.data.getVar() gets a bit old in a large event handler, and it means a
simple handler has to be modified if switching between an event handler
(e.g. RecipeParsed) and anonymous python. I think it would make sense
to restore the 'd' convention here to align with python elsewhere.
It'd just be a convenience, d==e.data, to avoid the common pattern of
setting it at the top of the event handler.
"""
I couldn't find a way to inject 'd' via locals/globals due to the use
of a function parameter so this left __builtins__ as the only way
I could find to make this work.
[YOCTO #7668]
(Bitbake rev: 44ac81e5281fb62ad00e2f79a9d754118ea62526)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than just use d.getVar(X), use the more explict d.getVar(X, False)
since at some point in the future, having the default of expansion would
be nice. This is the first step towards that.
This patch was mostly made using the command:
sed -e 's:\(getVar([^,()]*\)\s*):\1, False):g' -i `grep -ril getVar *`
(Bitbake rev: 659ef95c9b8aced3c4ded81c48bcc0fbde4d429f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The metadata can potentially use such an event to clean up any
"unreachable" data, solving several problems we currently have
where obsolete data may continue to exist in the shared areas.
(Bitbake rev: c5e6f929f3d5eeb7954660dea62611c58b795ff8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we're not going to change the value of FILE, or we know it isn't
going to have changed (ext == bbclass), don't set FILE.
This avoids messy looking history of the variable as well as optimises
parsing speed slightly.
(Bitbake rev: 88e4600aa66dda2e6c807f9d97af8982bcd8817b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake logger now sets a default project if the TOASTER_BRBE
or TOASTER_PROJECT Bitbake variables are not set.
This a necessary step in getting all builds under a project,
as to unify the MANAGED and interactive modes.
Other small fixes are included, related to the size of the
fields in the database.
(Bitbake rev: 5e0bf388f4e5c1cc493ac8264785e631bad2f672)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This refactoring brings the "local_path" of the
layer from the Layer object to the Layer_Version object, which
is more appropriate as different checkouts of the same
Layer may live in different directories.
This enables us to store Recipe file paths relative to a Layer_Version
at all times, aleviating the need to store full file paths in the
database. We also turn the prefix of the path (e.g. virtual:native
path name space) into a pathflag field.
In turn, this solves the problem of mis-identification of tasks based
on the recipe file paths, since we can also match the namespace of the
file paths on the recipe files.
[YOCTO #7594]
(Bitbake rev: ec43dc569e370767c709dec225cbee0c99151c19)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
At least in theory, the order the keys are expanded in can make
a difference, particularly if there is key overlap.
We also want to ensure that any underlying base key is processed
before any overridden version of that variable (FOO before FOO_x)
which helps the update_data removal code I've been testing.
(Bitbake rev: 863b6add24c211d64ba7931647084321f2d65889)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding a dependency when errors occur accessing a file when calling
handle() is not the correct thing to do. THe handle() code calls
resolve_file() which can raise an exception without ever touching
"fn" itself, it has also already marked all the dependencies correctly.
This leads to bogus files being resolved to the local cwd and
hence triggers reparses for no good reason.
The solution is to simply remove the bogus dependency.
(Bitbake rev: 366af3be1cffd64e4a79c15990c1e05869022c14)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst bitbake has done this for a long time, the behaviour of resolving
class files against cwd is not desirable. This can be seen during
base configuration parsing when looking for base.bbclass where a dependency
on cwd is added. If cwd then changes, the cache is invalid and triggers a
re-parse.
The only real option is to drop this entry and if files can't be found, we
fix BBPATH in the cases where it needs fixing. I didn't find any in the
random selection of layers I tested parsing locally.
(Bitbake rev: 508aad9d5db7e51328b1fd6ee53b4bc3720a30b7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the cooker parsing processes each dump an individual profile
which is ok, but means absolute numbers of function calls for a given load
can be tricky to determine as parsing of recipes may go to different pool
threads on different runs.
This change collects up the individual thread parsing results and processes
them into one profile output. The profile processing function in utils
needed tweaks to allow this to work.
(Bitbake rev: d3d2541aacd1ea560da0d8b25a3ea3f0563dee70)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This ensures basic key expansion works and that overlapping
keys generate a log message.
(Bitbake rev: ed5a8954ac923eda9750a636c5bb5b95ffce664f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows us to write tests which ensure a particular action generates
a particular log message.
(Bitbake rev: b30ee0aba51a35a194a4338b988f93ece1ed281c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was incorrect, it *must* be done next to the rename as a previous
may overlap with the current one and we need to detect this case.
I'll add a test case to better catch this problem in future.
(Bitbake rev: 5e9d0911cd86e980ab310cc0d79c9383fbc2c844)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This comparison is interesting even in the case of empty vales. Enabling
this warning actually found a bug in the metadata in avahi. Make the
code handle None specifically and also remove the dead code path in the
second if statement.
(Bitbake rev: a4cd4c56284812efb2a2bc0c8667ddad073f8e94)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By doing this we can take advantage of the expansion cache before
starting write operations on the data store.
(Bitbake rev: 702b42a47904f2378dd819e7463b3206883c2651)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the number of task dependencies change you currently get
a traceback when using diffsigs.
(Bitbake rev: c6798b431571aae18bb8699ac6e3ec75b731d719)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Like the Git fetcher, the Mercurial fetcher shouldn't expect recipes to
provide a checksum. As described [1], recipes using a mercurial
fetcher that don't provide a checksum will fail in a the repository has
previously been downloaded and archived.
Credit to Rafaël Carré for figuring out the bug.
[1] https://www.mail-archive.com/openembedded-devel@lists.openembedded.org/msg41328.html
(Bitbake rev: 2df35a25b4968f64adfa673d5b73442c1a30829d)
Signed-off-by: Ash Charles <ashcharles@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
gitpkgv_revision returns a sortable revision number that can be used
in the PKGV variable for example. To mimic meta-openembedded gitpkgv
behaviour to provide a sortable revision numner, one could set the
following:
PKGV = "1.0+${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"
This would yield a package version like "1.0+69+fb5eb80".
(Bitbake rev: 989c08f62aff7b707c25c692c23284f16506b7bc)
Signed-off-by: Mike Looijmans <mike.looijmans@topic.nl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The idea here is to support package version numbering similar to gitpkgv in
meta-openembedded. This commit is the first step towards such functionality.
The original plan was to add a "get_pretty_srcrev" method to the fetcher, as
per Richard's suggestion [1]. While writing this, I noticed that it would
become a copy of get_srcrev with only two lines changed. So to create something
more Pythonic than a boolean argument and conditionals around the calls to the
fetcher's sortable_revision, I just made the method to be called on the fetcher
an argument to the method. Defaulting to 'sortable_revision' prevents affecting
existing code.
Now if the git fetcher were to implement, say 'gitpkgv_revision' one could
set the following in a recipe:
PKGV="1.2+${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"
and this would yield the same result as gitpkgv's GITPKGV variable.
See for the discussion leading to this change:
[1] http://lists.openembedded.org/pipermail/openembedded-core/2015-January/100345.html
(Bitbake rev: 2f1f4483493cc290f5d2c07f9906e90eaea2f4c1)
Signed-off-by: Mike Looijmans <mike.looijmans@topic.nl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Pass proper repository url without arguments after a semicolon.
Executing checkuri on a rule with git repository in SRC_URI does
not report errors when working offline because wrong repository
url is passed to the ls-remote command. For example
"bitbake -c checkuri glibc" command executes:
"git -c core.fsyncobjectfiles=0 ls-remote git://sourceware.org/git/glibc.git;branch=release/2.21/master"
command in a shell subprocess to determine if url is valid.
Shell subprocess executes in fact 2 commands:
"git -c core.fsyncobjectfiles=0 ls-remote git://sourceware.org/git/glibc.git"
and
"branch=release/2.21/master"
First one returns 127 or 128 depending on error but second one
returns 0 because it is just env variable setup. Therefore we're not catching
connection error.
[YOCTO #7558]
(Bitbake rev: efa44d04137977f883db4a643b0f774e91514722)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix several bugs and add some useful enhancements to make this into a
more generic metadata editing function:
* Support modifying function values (name must be specified ending with
"()")
* Support dropping values by returning None as the new value
* Split out edit_metadata() function to provide same functionality
on a list/iterable
* Pass operation to callback and allow function to return them
* Pass current output lines to callback so they can be modified
* Fix handling of single-quoted values
* Handle :=, =+, .=, and =. operators
* Support arbitrary indent string
* Support indenting by length of assignment (by specifying -1)
* Fix typo in variablename - intentspc -> indentspc
* Expand function docstring to cover arguments / usage
* Add a parameter to enable matching names with overrides applied
* Add some bitbake-selftest tests
Note that this does change the expected signature of the callback
function. The only known caller is in lib/bb/utils.py itself; I doubt
anyone else has made extensive use of this function yet.
(Bitbake rev: 20059e4d5ab9bf0f32c781ccb208da3c95818018)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Fix unchanged assignments being dropped if other lines changed
* Fix not passing variable name from single-line assignments to the
function
* Fix not trimming the trailing quote from values
(Bitbake rev: 0b0c82f49cf2de887967d305768cbd95314bb171)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In certain contexts it can be useful to find the layer that a file (e.g.
a recipe) appears in.
Implements [YOCTO #7723].
(Bitbake rev: 3bf9c8830c5d5eea5502230d5af84ebd87ad5849)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add some tests of mirrors or mirrors to the fetcher unittests.
(Bitbake rev: e33d82bc10283d533f928836d56a6f0af80ea5c1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
First, when building mirror urls we don't do any fetching so we should never
be calling clean functions.
Currently, if a mirror url fails, we don't process it further to see
if there are any mirrors of the mirror.
We should do this even when the mirror url fails, else we may miss out
on valid/useful mappings, particularly in the case of file:// urls.
(Bitbake rev: b7fd3ec9994f664b17fc86423e6e7afac07e897b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow GIT_SMART_HTTP to be passed through to the fetch command so that
servers that cannot use GIT_SMART_HTTP can be used by the fetcher.
(Bitbake rev: e5c97a85bed0436d48eeaac2e32962cfb5371d2f)
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The URL is not sent when _latest_revision generates and exception.
When performing the sanity checks it is not possible to know the URI that failed.
This add the URL when latest_revision generates an exception.
[YOCTO: #7592]
(Bitbake rev: 9f2115b07a55cb14e4a74dc6fbd3707c28a234d0)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add some sanity checks on the parsing state engine when returning data
so that incomplete functions raise parse errors.
This means a recipe doing:
do_somefunction {
echo 1
VAR = "1"
will now raise a ParseError. To get the right file/line information,
__infunc__ was changed to a list.
[YOCTO #7633]
(Bitbake rev: 6b54a72638f57882d4fd5aab96b2752a09e065af)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fetcher calls os.chdir() in a number of places, which can affect
other tests (since the directory it changes into gets deleted) - let's
just put the current directory back to where it was when we're done.
(This fixes bb.tests.Path.test_unsafe_delete_path failing if it was run
as part of a full bitbake-selftest run, where the fetcher tests get to
run before it.)
(Bitbake rev: b1653855c74f86909c9f329ed6d2b10391c28395)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you're interested in using the checked out repository for development
(e.g. in OE with devtool) then you ideally want the origin remote to
point to the repository it was fetched from, so just set that after
cloning.
(As part of this I did a minor refactor so we have one function to
generate the repository URL, which was already in two places.)
Fixes [YOCTO #7756].
(Bitbake rev: 80ecd1c54d4c748cee3a7ce0d64013a346e7671e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases we need to check specifically for siginfo files, in
some cases we need to check for the actual sstate objects themselves.
Therefore make this a parameter to the function. A fallback to the
previous function style is maintained for now.
(Bitbake rev: 18d3a03e1b07c98b2dce46eb94f30de1a2b4320b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BB_ALLOWED_NETWORKS is a list of hosts that the fetcher will be allowed
to use when BB_NO_NETWORK is not set.
If BB_NO_NETWORK is set, then networking is still disabled.
If BB_ALLOWED_NETWORKS is not set, the behaviour remains the same as
today.
If BB_NO_NETWORK is NOT set, and BB_ALLOWED_NETWORKS is configured, then
only the hosts in the list are usable by the fetcher.
eg:
BB_ALLOWED_NETWORKS="yoctoproject.org git.gnu.org"
The fetcher will be able to download from yoctoproject.org, git.gnu.org,
but not ftp.gnu.org or any other hostname that is not in the list.
There is also limited support for wildcards on the beginning of the
hosts, so BB_ALLOWED_NETWORKS="*.gnu.org" with match git.gnu.org and
ftp.gnu.org as well as foo.git.gnu.org
(Bitbake rev: c7263096ba31ba45daeeb9de90c1cb9ebef24a28)
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use hg clone and hg pull to copy the source into the build
directory rather than taring up the cloned repository and
untarring in the destination.
This allows submodules to be cloned. While here, make the default
behaviour keep the hg scm data to match the behaviour of the git
fetcher.
(Bitbake rev: f002b1ca80cb542a4ed0c06c53c914cd5e076565)
Signed-off-by: Daniel McGregor <daniel.mcgregor@vecima.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Setting BB_ENV_EXTRAWHITE to include GIT_SSL_CAINFO is enough to
get "git ls-remote" to work for recipe parsing but it is not enough
to get the fetcher to clone properly.
This is because the fetcher has its own idea about what variables
should be exported in the environment for some operations.
It is desirable to use alternate CA Bundles for internal testing prior
to using public keys for https, so we should allow the GIT_SSL_CAINFO
to pass through.
(Bitbake rev: 40ff92282bbf32cf644b021bf7cbb1f393dbb856)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Moving the code to close the build and instantiate a fresh
buildinfohelper object to BuildCompleted event, as the
CommandCompleted/Failed/Exit events come in too early.
(Bitbake rev: af63abe88327fd5c1b3d7c00a84d9e408ef23285)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch modifies the toasterui to properly return the exit
code based on the errors found in the toaster itself.
The upload event file API call will not delete event logs for which
toasterui showed an error. This will facilitate debugging.
Minor enhancement in the buildinfohelper to reduce the number
of lookups on unknown layer objects (prevented testing of the patch).
(Bitbake rev: 1ddd6a9e4280a4adf971132ff1fe7ec9b3252905)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Introduced build mtime cache structure. Reset it before the build
to prevent bitbake from crashing when build/tmp/stamps hierarchy
is removed.
[YOCTO: #7562]
(Bitbake rev: f8590547a198a78334debdf14bf40acb50c22ecc)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake program and core versions must match.
Moved __version__ from main.py back to bin/bitbake.
Implemented check for version match in bin/bitbake.
(Bitbake rev: 2fe7d8c574ddf6a30278cff1a5a5c4089dc56d6d)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
tbs
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some error messages were lost because BBMainException was
inherited from bb.BBHandledException. When bb.BBHandledException
is processed error messages are not printed as they suppose to
be printed before raising this exception.
Stopped to inherit BBMainException from bb.BBHandledException.
Handled BBMainException in bin/bitbake and printed error message
to the stderr.
(Bitbake rev: c8e2a40c4e9865ebef9936d23644f2602a5c90f5)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add some very basic safeguard against recursively deleting paths such
as / and /home in the event of bugs or user mistakes.
Addresses [YOCTO #7620].
(Bitbake rev: 56cddeb9e1e4d249f84ccd6ef65db245636e38ea)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Implemented processing of ~ in bblayer's paths if HOME
environment variable is approved.
(Bitbake rev: 3b8a656d3ccb543c32696229184ebf12237ad38e)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the subpath parameter to the git fetcher ends with a trailing '/',
bb.utils.prunedir() will be called on '/'...
Fixes [YOCTO #7620].
(Bitbake rev: 380a3fb372c8b0a53dd7528562e6e7a222dc76ef)
Signed-off-by: Anders Darander <anders@chargestorm.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This ensures that _append, _prepend, overrides, etc are functional when used
on sstate variables (e.g. SSTATE_DIR).
[YOCTO #7564]
(Bitbake rev: 2e683c25b856b431198573f7f352d841587275e6)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
self.appendlist is a dict and as such unordered. This can lead to cases
where appends with different names (e.g. x_%.bbappend vs. x_123.bbappend)
can be reordered in application which in turn reorders the variables
that those bbappend files might touch. Reorderd variables changes the sstate
cache signatures causing real world issues.
To avoid this, use a list for the append files instead.
This patch is conservative and just adds a new data structure alongside
the existing one and uses it to resolve the core issue. Later patches
(post release) can handle some of the wider but less problematic ones
(e.g. issues in bitbake-layers flatten).
[YOCTO #7511]
(Bitbake rev: f980f060cd0d1e7fe5011f3c325c1b254f05eccf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since toasterui acts as the user-facing UI, we need
to run sanity checks in order to let the GUI display proper
warnings and stop the build if something is wrong.
(Bitbake rev: 260dd77fa771ae3b777134f4178d344e96b6f3d6)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch prevents tracebacks and instead logs exceptions
that may happen during event processing.
[YOCTO #7216]
(Bitbake rev: 0412631fb4a15ff42bf5ee46a77920fa558ae358)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Moved most of functionality of bin/bitbake to lib/bb/main.py
to be able to call bitbake from python code.
(Bitbake rev: d377f7f88d73f4e5d2dffef03d6acee809827ac6)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch fixes the recipe and layer identification by path
when Toaster uses relative paths.
(Bitbake rev: a92bb33a3ceacab2bfee9df1c39a202832866970)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We improve logging and signalling of errors in the
toaster_ui.log to facilitate debugging on remote systems.
(Bitbake rev: 3cd248f99b90367bd41aab81e255fc1912434890)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It used 5 spaces as the indent.
(Bitbake rev: 162d35ed53d34b28b153adf643044e7f105fcff1)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We modify the toasterui to log relative recipe paths
in order to maintain consistency with data fetched from
the layer sources.
(Bitbake rev: 253d69e88fd68729196ad43c15e8733527d76198)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In build mode, instead of creating our own layer objects,
we identify the layer objects that the build system set up.
[YOCTO #7378]
(Bitbake rev: 22962b540ace6868cb357c0fd13f01ffd24449c4)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Every time the bitbake show versions command (bitbake -s) is run it creates
a 100k log file.
The consolelogfile is disabled for show environment and disabling show
versions would make the behaviour match.
(Bitbake rev: dee0ba94e39ea49c1e9261a5e8932356e3bb7c57)
Signed-off-by: Rob Woolley <rob.woolley@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Any exceptions that occur in calls to logging methods are automatically
suppressed, including exceptions due to broken pipes.
However, the knotty summary messages are printed directly to stdout, which
means that any broken pipes will cause an exception traceback in python.
By wrapping the summary section in a try / catch block we can check for
IOError exceptions caused by broken pipes and let them pass.
(Bitbake rev: 146e7e157f97b676858ecff583dd53800d997253)
Signed-off-by: Rob Woolley <rob.woolley@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The recent inotify changes are causing a 100% cpu usage issue in the
idle handlers. To avoid this, we update the idle functions to optionally
report a float value which is the delay before the function needs to be
called again. 1 second is fine for the inotify handler, in reality its
more like 0.1s due to the default idle function sleep.
This reverts performance regressions of 1.5 minutes on a kernel build
and ~5-6 minutes on a image from scratch.
(Bitbake rev: 0e0ba408c2dce14a0fabd3fdf61d8465a031495b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
[YOCTO #5571] -- https://bugzilla.yoctoproject.org/show_bug.cgi?id=5571
The following workflow (whether accidentally or deliberately) would
previously not result in a checksum error, but would be helpful to do
so:
- Write a recipe with correct checksums
- Fetch the sources for this recipe using bitbake
- Change the checksums
Since the bitbake fetcher writes a done stamp after the initial download
and does not verify the checksums again (even if they are changed in the
recipe afterwards), the change of checksums is silently ignored.
Fix this without the overhead of computing the checksums from scratch on
every do_fetch by storing them in pickled format in the done stamp and
verifying that they still match those in the recipe.
(Bitbake rev: fbd4a0d422cf7f43db2f9eab2e47c41246a9031e)
Signed-off-by: Clemens Lang <clemens.lang@bmw-carit.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch makes sure to delete an idle function that
raises an exception for the xmlrpc server.
The counterpart functionality in the process server was
added with:
commit db50630948.
duthor: Richard Purdie <richard.purdie@linuxfoundation.org>
Date: Wed Aug 20 22:31:06 2014 +0000
bitbake: process: Deal with infinite looping of the server
This patch fixes
[YOCTO #7316]
(Bitbake rev: e7c9a6788d969c901fd6394416ac3936e62c4c72)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch fixes a bug where if the build is force stopped,
subsequent clients cannot connect to the server due to
unnecessary limits on setFeature.
Additionally, we make sure that the state is properly reset
even if the BuildCompleted event firing excepts for some reason.
(Bitbake rev: 0b66b05169688aa4ddc4c54d175bb961b2f27fec)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We roll back the client connection if some error happens,
like during setFeatures, as to leave the server accessible
to other clients.
(Bitbake rev: 4e4a2ee2f05f8741b2e09263e328420363975b02)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The inotify facility monitoring changes to the config files
could be overwhelmed by massive changes to the watched files
while server is running.
This patch adds verification the notification watches to the
server idle functions, in addition to the cooker updateCache
command which executes only infrequently, thus preventing
overflowing the notification buffer.
[YOCTO #7316]
(Bitbake rev: 996e663fd5c254292f44eca46f5fdc95af897f98)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some databases, notably the SQLite3 adapter, force autocommits even if
the autocommit is turned off. The behavious is tracked in this bug:
http://bugs.python.org/issue8145#msg109965
Django refuses to work with autocommit off in this case, so we have
to take the same precautions when using manual transaction support.
[YOCTO #7363]
[YOCTO #7365]
(Bitbake rev: 90231ab63a129fa344d461c2911898ea0f07f206)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch fixes a problem where set sstate scene tasks
were not identified, causing cache attempt not being recorded.
[YOCTO #7223]
(Bitbake rev: 8a326a9a5a08981f1b7960e02fdb8a9436db16fb)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch disables autocommit for inserting build data,
effectively updating all build data in a single transaction.
This is a purely performance improvement patch, as the transaction
will always be commited.
Similar manual transaction handling in the layer source update
method. Added feedback messages during update method.
[YOCTO #7140]
(Bitbake rev: 3978c819e797f857235499a4b8ec238134f1c028)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch brings in cooker log saving and proper download links.
* toasterui will now write the cooker log file if running in managed
mode
* the BuildRequest has a new state, REQ_ARCHIVE, indicating that the
build is completed, and the artifacts are ready to be grabbed
* the runbuild test execution commands will gather needed artifacts,
and save them to a storage directory selected during Toaster setup.
* the build dashboard, project builds and all builds pages have
permanent links for the cooker log
[YOCTO #7220]
[YOCTO #7206]
(Bitbake rev: fad80e36c9da663b000cdf2cb3c75440c6431d84)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add set of PN in data because now latest_versionstring use it for
validate version directory searching.
(Bitbake rev: 2e4a03db967ac1459b2764108fc54c4566a7e371)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix sustition for rc, beta and alpha releses from -N to N weight.
(Bitbake rev: 63a9e60a6d80cfd2693ec1a6359785dc19f98e1f)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for scan every version directory using _check_latest_version
makes code more robust because sometimes upstream projects publish
new directories without files, causing don't find version.
To support this new behaviour remove _check_latest_dir and replace for
_check_latest_version_by_dir,
(Bitbake rev: 1a75b3707743c32eec9d2cf566fb6bbea9f73784)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add _check_latest_version_by_dir this function provides support
for scan every directory newer than current dir in order to get
latest_versionstring, example:
http://somedoamin.com/project/v2.1/http://somedoamin.com/project/v3.0/
Change return of _vercmp from True/False to -1/0/1 to provide test
when current directory is equal to newer directory this helps to
scan the same directory to get minor versions, example:
http://somedoamin.com/project/v2.1/project-v2.1.2.tgzhttp://somedoamin.com/project/v2.1/project-v2.1.6.tgz
(Bitbake rev: 5f7c5eb218a221165f59a0f4dd48d2d97f756193)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to reduce code duplication now compile package_regex in
_init_regexes instead of make this decision at _check_latest_version,
(Bitbake rev: e7284e3ad0e7dd91ed59dfbf8450ef62e89c7e54)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for get group only if exist in regex, this enables to use
this function in _check_latestversion regardless if the regex is generic
or specified by REGEX_URI.
(Bitbake rev: 1127af5b8c458929c4685b0326f86870ed09442e)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've had versioned dependency support in LAYERDEPENDS for quite a long
time, but I can say with pretty good certainty that almost nobody has
used it up to now because it was too strict - the specified version had
to exactly match the version in your configuration or you would get an
error; there was no "greater than or equal" option, which is usually
what you will want given that LAYERVERSION does get bumped from time to
time.
However, users mismatching layer branches and then having their builds
fail later on with some incomprehensible error is still a pretty common
problem. We can't simply use the git branch because not everyone is
always on a branch and the branch names don't always match up (and
that's not an issue). To provide a practical means to address branch
mismatching, I have reworked LAYERDEPENDS version specifications to use
the more familiar "dependency (>= version)" syntax as used with package
dependencies, support non-integer versions, and clarified the error
message a little. If we then take care to bump the version on every
breaking change, it is at least possible to have layers depend on these
changes when they update to match; we can now even support a major.minor
scheme to allow retrospectively adding a version limiter to old branches
when a new branch is created and yet still allow the old branch minor
version to be bumped if needed.
Fixes [YOCTO #5991].
(Bitbake rev: 408be9cdf2b1e32e64ea488d8051a546fb54c144)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This scheme is used for versioning recipes that are pre-release (alpha,
beta, etc.) within OpenEmbedded, so add some tests to ensure the
appropriate comparison results still hold true.
(Bitbake rev: 3a9eefe27f29a4593d6298f0427ac5f3e9183377)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We really want an error rather than the version to just be silently
skipped when the operator is missing (e.g. "somepackage (1.0)" was
specified instead of "somepackage (>= 1.0)".)
(Bitbake rev: b6dc946f477adc40d68da16e2f2580cb3b4a10db)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake -g -u depexp <package> segfaults when DISPLAY is not set
properly. Fix it with a proper check.
[YOCTO #7299]
(Bitbake rev: f35e9bd7b59c180fe9a3d9177efb57b92d9cd373)
Signed-off-by: Maxin B. John <maxin.john@enea.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch improves the logging facilities for toaster in order
to help diagnose bugs that happen on user machines.
The logs are stored now under "/tmp/toaster_$$" where $$ is a
PID-based unique identifier. On shutdown, toaster will automatically
erase all logs unless errors are listed in the log file.
On error, Toaster provides suggestions on what to do.
This patch includes a minor fix found as a result of logging
improvements.
(Bitbake rev: 8a8248f7b7e30469f592e2f8adbf6ce21e8685c5)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Given these assignments:
TEST="a b c d"
TEST_remove = "b d"
TEST evaluates to "a c". However, if the _remove override is given as a
variable:
TEST="a b c d"
FOO = "b d"
TEST_remove = "${FOO}
TEST evaluates to "a b c d", because when FOO is expanded it isn't split into a
list.
Solve this by splitting all members of removeactive once they've been expanded.
[ YOCTO #7272 ]
(Bitbake rev: 207013b6dde82f9654f9be996695c8335b95a288)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The _remove operator isn't working correctly when used with a variable that
expands to several items, so add a test case to exercise this path.
(Bitbake rev: cb2a62a5fbffb358528a85b46c1fc6383286cb9d)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These only have one instance created which means your subsequent datastores
can contain echos of previous ones. Obviously this is not the behaviour
we want/expect. It doesn't affect bitbake too badly as we only have one
datastore, it does massively potentially break our selftests though.
Thanks to Tim Amsell for pointing out the now obvious problem!
(Bitbake rev: 9facf3604759b00e8fe99f929353d46f8b8ba5cb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
ToasterUI filters build artifacts by extension in order
to determine if a build artifact is an image or not.
Using IMAGE_FSTYPES for this purpose is not correct as
the varible value holding image extensions is just a coincidence.
So we just look if the filename contains the "rootfs" magic
string, which is a pretty good approximation.
[YOCTO #7213]
(Bitbake rev: b11e8bd626e0212ee72914529c3d92d1dd718674)
Signed-off-by: Alexandru Damian <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The get/set_taskdata functions are now part of the API of the class,
ensure they exist in the base class definition so the noop handler
works.
[YOCTO #7233]
(Bitbake rev: 9b5b1bd7d77e3f5886f6c557d3b750de1f6d6025)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Users expect operators like _remove to work on this variable. We need
to use expanded_data to ensure this happens correctly.
[YOCTO #7135]
(Bitbake rev: cc4c8478fc547ea0ebf827a8d319496b39f25684)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
event_data would be better named expanded_data, then we can widen its scope
to other places in cooker where we need to access an expanded data store.
We certainly don't want multiple expanded data stores.
(Bitbake rev: 1a3c1c9203e1a1452314954f1cfd771e5c1ce89b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was a report that bitbake -e | less would use 100% cpu when it shouldn't
really. The issue appears to be a bogus file descriptor in the select call. We
shouldn't be blocking if there is event data pending to a *reader* from server
context.
[YOCTO #7138]
(Bitbake rev: 8f166e1a0f3574ae7d1e917a8bb403b87bad15bf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add a call that saves any queued events to the build
[YOCTO #7021]
(Bitbake rev: 4f5b19d453da64749affc1c27ec51b013bedc71a)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When built package regex for use in searching upstream version in sites,
the package name need to be escaped to avoid usage of special regex
char.
For example when search for gtk+, '+' need to be escaped.
(Bitbake rev: 1aa1de4b0c5bd34466e04844bbc371933736be59)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Trap the ENOSPC error and translate it into a human readable error
message, which is good for debugging.
(Bitbake rev: 2b084dff6ff0d274fbbf7ab07022507f7249e427)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We currently add crazy numbers of watches on files. The per user limit is 8192
by default and on a system handling multiple builds, this can be an issue.
We don't need to watch all files individually, we can watch the directory containing
the file instead. This gives better resource utilisation and better performance
further reverting some of the performance regression seen with the introduction
of pyinotify.
(Bitbake rev: a2d441237916a99405b800c1a3dc39f860100a8c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the environment changes, we need memory resident bitbake to adapt to those
changes. This adds in functionality to handle this alongside the configuration
option handling code. This means that the common usage:
MACHINE=X bitbake Y
now works with the memory resident server.
(Bitbake rev: 4d1343010da757a0c126bc22475354da44aaf8e3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tweak the comments on get_srcrev() to better describe its function.
(Bitbake rev: b4d40f1ac7b32990c456cce261f99a5a157b5ae5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A bug slipped in the toaster ui that prevented saving of
build configuration despite the data being retrieved
from the server. This patch fixes the shaming mistake.
[YOCTO #7117]
(Bitbake rev: 8118f465b9f87c66b2a741008f69198ac5fea901)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The import statements here are plain bizarre. Remove them, tweaking
some of the function calls to match current practices. I can't find any
reason these old imports are as they are.
(Bitbake rev: 4c2f1fe51a13ddc97e518327714292af46b9e1ab)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Looking at this function I had no idea what oldfn was, I doubt anyone
else would either without looking up what the caller does. "parentfn"
would seem a more appropriate name so rename it.
(Bitbake rev: fc70ed596703a1aa954223b169d4ad51193a6ec1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Benchmarks show that the introduction of pyinotify regressed
performance. This patch ensures we only call the add_watch() function
for new entries, not ones we've already processed which does improve
performance as measured by "time bitbake -p".
This doesn't completely remove the overhead but it does substantially
reduce it.
(Bitbake rev: 493361f35f6cc332d4ea359a2695622c2c91a9c2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We try and add watches for files that don't exist but if they did, would influence
the parser. The parent directory of these files may not exist, in which case we need
to watch any parent that does exist for changes. This change implements that fallback
handling.
(Bitbake rev: 979ddbe4b7340d7cf2f432f6b1eba1c58d55ff42)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Memory resident bitbake has one current flaw, changes in the base configuration
are not noticed by bitbake. The parsing cache is also refreshed on each invocation
of bitbake (although the mtime cache is not cleared so its pointless).
This change adds in pyinotify support and adds two different watchers, one
for the base configuration and one for the parsed recipes.
Changes in the latter will trigger a reparse (and an update of the mtime cache).
The former will trigger a complete reload of the configuration.
Note that this code will also correctly handle creation of new configuration files
since the __depends and __base_depends variables already track these for cache
correctness purposes.
We could be a little more clever about parsing cache invalidation, right now we just
invalidate the whole thing and recheck. For now, its better than what we have and doesn't
seem to perform that badly though.
For education and QA purposes I can document a workflow that illustrates this:
$ source oe-init-build-env-memres
$ time bitbake bash
[base configuration is loaded, recipes are parsed, bash builds]
$ time bitbake bash
[command returns quickly since all caches are valid]
$ touch ../meta/classes/gettext.bbclass
$ time bitbake bash
[reparse is triggered, time is longer than above]
$ echo 'FOO = "1"' >> conf/local.conf
$ time bitbake bash
[reparse is triggered, but with a base configuration reload too]
As far as changes go, I like this one a lot, it makes memory resident bitbake
truly usable and may be the tweak we need to make it the default.
The new pyinotify dependency is covered in the previous commit.
(Bitbake rev: 0557d03c170fba8d7efe82be1b9641d0eb229213)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When urls ends with trailing slash os.path.basename return "" [1]
and built urldata.localpath only with DL_DIR, it causes that
donestamp is built as DL_DIR + '.done' and seems that ssh resource
was already download.
[YOCTO #6448]
[1] https://docs.python.org/2/library/os.path.html#os.path.basename
(Bitbake rev: 47992591349bab2c12741b937096e41085399087)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the code ignores lightweight tags which has caused some user
complaints. We can't put the right search list in place easily since
the results don't come back in a good order, head happens to sort
before tags.
In the end I refactored the function so we get the complete list of
remotes and then we can filter it ourselves in the order we chose,
including checking for light weight tags, preferring the proper ones.
Hopefully this resolves the issues people have been seeing.
[YOCTO #6881]
(Bitbake rev: 07ad307065bb15a48f0015b9e4a643201abdc283)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A dash character is illegal in function names in sh (but not bash). Since
our shell tasks run under sh and the shell parser is sh based, EXPORT_FUNCTIONS
won't work with class names containing a dash.
We can't change sh, we can ensure the user is warned about the problem
straight away though.
[YOCTO #7006]
(Bitbake rev: 86704281b79e524dccccc88cbf996b299b33bae2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if you reference a file url, its checksum is included in the
task hash, however if you change to a different file at a different
location, perhaps taking advantage of the FILESPATH functionality, the
system will not reparse the file in question and change its checksum to
match the new file.
To correctly handle this, the system not only needs to know if the
existing file still exists or not, but also check the existance
of every file it would have looked at when computing the original file.
We already do this in the bitbake parsing code for class inclusion. This
change uses the same technique to log the file list we looked at and
if files in these locations exist when they previously did not, to
invalidate and reparse the file.
Since data stored in the cache is flattened text, we have to use a string
form of the data and split on the ":" character which is ugly, but is
an internal detail we can improve later if a better method is found.
The cache version changes to trigger a reparse since the previous
cache data is now incompatible.
[YOCTO #7019]
(Bitbake rev: 6c0706a28d72c591f1b75b6e3f3b645859387c7e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases for cache purpoes we not only need to know which file
is going to be used but also which paths were considered. Add a
localpaths method which includes the history.
The core which() funciton already supports this, this just extends
the function to preserve the extra data we need. localpath becomes
just a special case of the case with history.
(Bitbake rev: ea5efeac5c1f7986666c979f789786f29fc1619a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the cooker is in an error state, we shouldn't continue to try parsing.
This fixes an issue where an invalid PR server is detected when bitbake
is started and ensures bitbake exits cleanly rather than hanging.
[YOCTO #6934]
(Bitbake rev: 294bb9cad294423d4f8998405ceff58655f12660)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a little crude as the usage basically ignores that you're doing
an append operation, but for a lot of cases it will be sufficient.
(Bitbake rev: 24a28205ab680b6cc645d97b76c9855920608229)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Fix add-layer error message when a layer is already in BBLAYERS
* Ensure we show an error message if we can't find BBLAYERS at all
(Bitbake rev: 1c743fd2103730e27699dd55efc6914d3b0c3702)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Hardcoding a temp directory is bad practice and leads to races between
the tests. There is no longer any good reason for doing this, drop it
and ensure the files get cleaned up correctly.
(Bitbake rev: 10a47b1ec7470c9e8c4ffe0bb35cdf6d1bb2ee2e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you copy the datastore, then delete a key, it should not exist in
d.keys(). This adds a test to cover the recently found data store bug.
(Bitbake rev: 16d5f40ad20fd08bf7a4d0e36200c739b5a9f59e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The versionstring tests hit the network so should only run when
network tests are enabled.
Also remove the print statement which confuses the test output and
add it to the test failure message instead.
(Bitbake rev: 6c046660cfc4fb3792a42aeafff91a13f68a2e89)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you copy the datastore, then delete a variable, it still shows up
in d.keys() when it should not. This patch addresses the issue.
(Bitbake rev: f28ee1bb03cb32d3757fbef67c9fbe143e3dadfa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to fix multithread usage of latest_versionstring moves package_custom_regex_comp
from class to method level because need to be defined by package.
Remove code for build url's with /download suffix because it's deprecated since you can
specify the download directory using package_regex.inc file.
(Bitbake rev: 231cae9f9b552ec6737795c098d1de426b5adcbc)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a BBFILE_PATTERN_IGNORE_EMPTY variable to allow ignoring the fact
that a regex specified in BBFILE_PATTERN for a particular collection
doesn't match any recipes. This will be used in OpenEmbedded in the
workspace layers created by "devtool" which may not always contain any
recipes (which is not cause for warning the user).
(Bitbake rev: 19c74aaa2836a88d666f0032452fac521689ab6b)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Add a generic edit_metadata_file() function to modify variable
assignments in any metadata file (conf, bb, bbappend) using a callback
for flexibility
* Add a specific edit_bblayers_conf() function to modify
conf/bblayers.conf and add and/or remove layers from the BBLAYERS
value within it.
(Bitbake rev: aa03a28b442549dd8ffe92ae4d6390f62202a76a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a nostamp task is depended on by a non-nostamp task, then we want the
signature of that task to change such that it re-executes afterwards.
This is an unusual situation, but we want this to work in OE in
externalsrc.bbclass so that compilation happens every time it is
requested.
(Bitbake rev: 73498afc3d45beede5b8f24a9acd523a1663b793)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Running bitbake inside make results in the exported environment variable
MAKEOVERRIDES="${-*-command-variables-*-}", which the shell chokes on
when trying to expand it. But of course, it probably shouldn't have been
trying to expand it in the first place -- so just escape the dollar
sign.
(Bitbake rev: 18cd0ce6a55c9065c3f1bf223b47d817b5efcd8f)
Signed-off-by: Richard Tollerton <rich.tollerton@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds a "-w/--write-log" option to bitbake
that writes an event log file for the current build.
The name of the file is passed as a parameter to the "-w"
argument. If the parameter is the empty string '', the file
name is generated in the form bitbake_eventlog_DATE.json,
where DATE is the current date and time, with second precision.
The "-w" option can also be supplied as the BBEVENTLOG
environment variable.
We add a script, toater-eventreplay, that reads an event
log file and loads the data into a Toaster database, creating
a build entry.
We modify the toasterui to fix minor issues with reading
events from an event log file.
Performance impact is undetectable under no-task executed builds.
(Bitbake rev: 1befb4a783bb7b7b387d4b5ee08830d9516f1ac2)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We update and add logs throughout the code in order to help
with development. The extra logging is turned off by default,
but it can be enabled by using environment variables.
All logging happens through the Python logging facilities.
The toaster UI will save a log of all incoming events if the
TOASTER_EVENTLOG variable is set.
If TOASTER_SQLDEBUG is set all DB queries will be logged.
If TOASTER_DEVEL is set and the django-fresh module is available,
the module is enabled to allow auto-reload of pages when the
source is changed.
(Bitbake rev: 10c27450601b4d24bbb273bd0e053498807d1060)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add a BuildArtifacts class to store data about files
discovered during the build process and not stored anywhere
else.
Small cosmetic changes in the toasterui.
Add model methods to return file path display data relative
to the build environment instead of absolute file paths.
[YOCTO #6834]
(Bitbake rev: bbe24d912869312d561be199b2c029b0c898e049)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After Ctrl+C is pressed to interrupt bitbake, it loops continually, running
at 100% cpu. This patch selects on the correct file descriptors resolving
the excess cpu usage.
(Bitbake rev: 497404e8484b7ca7c11e459bf0845642156eb677)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The shellshock patches changed the way bash functions are exported.
Unfortunately different distros used slightly different formats,
Fedora went with BASH_FUNC_XXX()=() { echo foo; } and Ubuntu went with
BASH_FUNC_foo%%=() { echo foo; }.
The former causes errors in dealing with out output from emit_env,
the functions are not exported in either case any more.
This patch handles things so the functions work as expected in either
case.
[YOCTO #6880]
(Bitbake rev: f28f37220e7787721a31b659521a1c44ebea92bf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you don't explicitly specify to use a global variable when doing an
assignment, you will be setting a local variable instead, which means
this function wasn't working at all. It explains some odd behaviour we
have seen in the layer index where event handlers were sometimes
bleeding into other contexts where they should not have been.
(Bitbake rev: ac45ea848901b0f6cd23087b662dde8ce9cd807e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using external tinfoil-based utilities, it is useful to be able to
turn off most of the event handlers; for example sstate_eventhandler
doesn't like being sent events for any recipe which has been skipped.
(Bitbake rev: 41236c28985a3b66b3737382a94e39dbf6480160)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we're fetching outside of the context of a recipe, it's handy to be
able to disable checksum functionality so you don't get a meaningless
warning about the signatures being missing.
(Bitbake rev: 49dbcfbc56a206964acc5de761bba31be0283ba1)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a function that allows executing a flat python function (defined
with def funcname(args): ...).
(Bitbake rev: 20e6939ebcb62e08a9a7ad586a915dfe368136a0)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The defaultval field is intended to be internal and the only use of that field
outside of data.py is to skip over it when iterating over a value's flags.
For clarity and convenience, rename the field to _defaultval so that it is
considered internal and not exposed through the data API.
(Bitbake rev: 2800958dadaa5c055ba21d52c98d842d360f0785)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if an invalid PR service is selected the server will error
with a traceback. This is because its set into the error state and the
setFeature code will then fail since its not in the initial state.
Modifying the featureset in the error state is acceptable, we just need
to ensure we don't trigger a reset, that would happen from whichever
code handles the error.
[YOCTO #6934]
(Bitbake rev: c52841445d8db8f84c4da34203b195fea5874247)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update test case for cups is needed because match only 2.0.0
versions see VERSION=2\.0\.0 in the previous string.
(Bitbake rev: 148aba30155f4de17f6c6fb9b4c37f08a2db202b)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some upstream sites put the name of the package in the body of href tags,
i.e. <a href="#43">somepackage-v1.4.10.tar.gz </a>.
(Bitbake rev: 4fa8542a8880d5749fbb3382706e011b012024d0)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Validate if package contain version string if not return the current
version cases for spectrum-fw and corpus recipes.
_check_latest_version return the latest version available don't
take into account the current version previous this only return
the upstream version if it greater than the current version.
(Bitbake rev: 91a7ac8c7f87f98e366585cf9720ec35b0790bae)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
package_custom_regex_comp is built with the current package name and
then used to search upstream version this reduces custom regex'es in
sites that have different packages in the same directory.
(Bitbake rev: a69d5d48c4bf9a3df3121cc4e69ba5d7e947ad8d)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Latest version string only try to find latest directory when REGEX_URI
isn't specified to avoid unnecessary processing and makes code easier
(Bitbake rev: afc33ec7cdb7d8ee3602a23fa973551ca5510ac4)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch brings in changes that allow a toasterUI coming in
from 'master' branch to record data from a 'daisy' or 'dizzy'
bitbake server.
This is needed to allow Toaster to record builds running
on older branch releases.
(Bitbake rev: 8d75e28e0688a6520311afce36543175f36910b3)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we have a managed build, we match the layers used for build
with the layers configured for project, as we know where the layers
are coming from
[YOCTO #6962]
(Bitbake rev: e02ec052a62cbc476bdac65cb7cea1167ce04781)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Changing ToasterUI to log toaster exceptions on a different level than
build errors.
Updating the build dashboard to show Toaster exceptions.
We add extra logging to console for exceptions.
Fixed a problem where packages database entries were created instead of
being looked up in the database, conficting with entries created to
satisfy dependency information.
Toaster now checks for invalid states at startup and performs needed
cleanups.
Removed loading reference to jquery-ui.min.css as we do not have this
file.
(Bitbake rev: 2378812bc24d433125fb940f110154f0ce638448)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
1) Run "bitbake recipe" in the terminal
2) Close the terminal while building
3) $ ps aux | grep bitbake-worker
There will be many processes, and they will keep the resources (e.g.,
memory), and won't exit unless kill or kill -9.
(Bitbake rev: 40d2ae0723de2bf5fee343faafb4afda40546839)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The oe.utils.contains function has been removed from OE-Core metadata
as the references for it has been replaced to use the
bb.utils.contains.
(Bitbake rev: 5cfdebe7a67dccc7552ff80c1ccc970e36d562df)
Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This state was renamed in bitbake 12e9d3 but this use of it wasn't changed.
[ YOCTO #5445 ]
(Bitbake rev: 196106100e2dedfacce95be0ee6def94c5a80f27)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
loginfo "op" was being set to invalid values in saveConfigurationVar it was
working because set is a python global but append is not. This replaces both
the "op" with strings and removes the crash when calling appendConfigurationVar
(Bitbake rev: 2a8e847de85546d43600a561f9c63aa36bd69222)
Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Improve the performance of data logging in toasterui.
We modify the data queries used to:
* cache searching in memory
* insert in bulk (i.e. multiple values per insert, where possible)
On development test rig (networked mysql), on no-op build,
time for data recording is reduced from 4:10 to 1:30 (minutes).
We also improve the logging, so it is easier to detect
toasterui errors.
(Bitbake rev: d42784432f927f58730caf80546c66772e0fec89)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Being able to query whether updated versions of a url are available
is useful, not least for the package reporting system. Since such code
is closely linked to the url type and the url itself, the fetcher
makes a locical place to contain this code.
For wget based urls this means taking upstream directory listings
and searching those for later versions, returning those that are
found.
The patch also adds unittests for this function so that if
improvements are made, the original test urls can be used
to evaulate the those changes.
This is based on code from Irina Patru <irina.patru@intel.com>.
(Bitbake rev: a8272e22b7819e0e8afd8e291d276f5f28fc0007)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Being able to generate a version string representing the most recent git commit
given git is useful, not least for the package reporting system.
This adds in a latest_versionstring method to the git fetcher
which allows users to query the latest version using ls-remote
and filtering the responses.
The patch also adds unittests for this function so that if
improvements are made, the original test urls can be used
to evaulate the those changes.
This is based on code from Irina Patru <irina.patru@intel.com>.
(Bitbake rev: f71c8c0354e87fed80bc845db6728e6e18ce9c4d)
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we split variables only at whitespaces, a slipped in tab will render
a value unremovable.
(Bitbake rev: 9f171ea755644ecd9d2b3d7ed13bf8ec09ec917a)
Signed-off-by: Stefan Müller-Klieser <s.mueller-klieser@phytec.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Similar to the last shared work task signature bug, we've found another
one. Looking at the improved output of diffsigs in this case:
runtaskdeps changed from [
'autoconf_2.69.bb.do_populate_sysroot:virtual:native',
'gnu-config_20120814.bb.do_populate_sysroot:virtual:native',
'libgcc-initial_4.9.bb.do_patch:virtual:nativesdk'
] to [
'autoconf_2.69.bb.do_populate_sysroot:virtual:native',
'gcc-crosssdk-initial_4.9.bb.do_patch',
'gnu-config_20120814.bb.do_populate_sysroot:virtual:native'
]
so we can get a different task hash since libgcc sorts before gnu-config
and gcc sorts after it. We could do with a way of fixing this, the best
I can come up with is to include a single parent directory. Since
recipes are never at the top of any metadata trees I've seen, this
should suffice for now.
I'm planning to burn the concept of shared work within bitbake
and do something at the metadata level in the 1.8 timeframe as its just
too fragile as things stand and hard to fix well.
(Bitbake rev: d753644c67d163f338f2bdc3d600203e8b1a5734)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is useful code to double check the computed checksum value if nothing
else. Might as well have it in tree.
(Bitbake rev: 54ecf96c6f031927ee2410f6efde4e16f19bbf66)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Order of runtaskdeps is important. If the hashes differ we should print output.
This is complicated by shared work where the filenames themselves can differ,
but the checksum should not.
This fixes a case where two different checksums could show no output with
bitbake-diffsigs.
(Bitbake rev: 40c95cb9def282dc88234cd72ff462d7a01e47c1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add improvements in data reading, following
issues discovered in testing.
- elapsed_time is now read from buildstats
- we add safeguards to not fail logging if the build was triggered
with a toaster_brbe configuration, but it's running in
1.6 mode
- added log markups for build finish to let other programs
known when the work is done.
[YOCTO #6833]
[YOCTO #6685]
[YOCTO #6887]
(Bitbake rev: 0b225035cefee3d3713a93f9a432e5e4d4e174f1)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This has always bothered me:
NOTE: Preparing runqueue
NOTE: Executing RunQueue Tasks
This patch changes the messages to be consistent.
(Bitbake rev: 72ac9f9227fbfb4dc8b933b357d21aa0e4060959)
Signed-off-by: Gary Thomas <gary@mlbassoc.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This makes it possble to fetch Gerrit review references which are
normally stored under refs/changes.
Please disregard previous patch with the same topic.
(Bitbake rev: 268e9c0c6830e8e621c418f20c2ca12dc840e48b)
Signed-off-by: Fredrik Svensson <fredrik.svensson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We avoid a race between the setting the TOASTER_BRBE variable
and reading the variable in toaster ui by supplying the variable
at server startup time through the toaster.conf post-read file.
Additional small changes are included, including marking the
build request with the environment id of where the build took place.
(Bitbake rev: 7c333350418c4140e6c988c5272940f8057d327d)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We change the saving of the build id to the build request as
soon is the build is created, as to allow for a consistent
display of build data while build is in progress.
(Bitbake rev: 9504ca6a69ba6da21f88b3cc77fa5910d886b6c3)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to consistently use LogMessage.INFO/WARNING/ERROR to make sure toaster knows
how to categories these rather than passing in the "raw" loglevel value
which in best case comes from python logging but worst case any value.
[YOCTO 6885]
(Bitbake rev: 3aa13bc1d8218c97c76581a895fa5f03ff807fbe)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
See http://www.nongnu.org/lzip/lzip.html for details on the compression
format.
(Bitbake rev: 9027b1273b5405c7269b013604ab417771b5eafe)
Signed-off-by: Koen Kooi <koen@dominion.thruhere.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Forcing strict to be a string, to avoid problems when performing comparisons
[YOCTO #6762]
(Bitbake rev: b8ed2098bdea2afd93ab4e3e1b834f3a31cb60de)
Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
hob was adding the redundant characters "\1" in SSTATE_MIRRORS variable. If
needed it is expected the user will add this instead so remove the code
that was doing this.
[YOCTO #6600]
(Bitbake rev: 73bf120062fc00c7e26dc4e77a7d140658d89daf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fixes urls of the form file://some/path/file;subdir=b. It also
adds in a couple of tests so we now tests these corner cases.
(Bitbake rev: 46306912a96444790efa9418d934dfdd36773ba1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Check if the 'subdir' parameter exists and assign it to 'destdir' so that
files are copied in ${WORKDIR}/destdir. This fixes urls that are of the form
file://a;subdir=b.
(Bitbake rev: 836a986b365eb9798563ec08d90b346596de7791)
Signed-off-by: Roxana Ciobanu <roxana.ciobanu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Writing a log that the filesystem isn't being monitored for inode usage just
confuses users who are not aware about the nature of inodes in their filesystem,
so don't say anything, just silently disable the monitor. In general this only
happens on filesystems which don't have a limit on inodes.
(Bitbake rev: ca93bc84ee5fb94a50c11c47e4d212d7da649e24)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I could apply the label "release_1.0" to a super project that contains
many sub projects. If my recipes have SRC_URI's that use that label but
grab different sub-folders, than there's a bug where the cached localfile
(tar.gz) will not be unique and reused at the wrong times.
SRC_URI = "p4://perforce::1234@//depot/SuperProject/MiniProjectAAA/...;label=release_1.0 \
p4://perforce::1234@//depot/SuperProject/MiniProjectBBB/...;label=release_1.0"
(Bitbake rev: 3b5b1703b77490116dda895b29737cea51a3d6a0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using options like -k, -f, -v and so on with the memory resident
server, they'd currently only be set on the initial values passed to
the original command. This ensures they now match those specified
on the commandline for the options where this makes sense.
To make this work, a command to update the options on the server side
is required so this is added.
[YOCTO #5292]
(Bitbake rev: 1c75cc4d0c8b606c1fe76e6bf60bf6a32298b105)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you have:
FOO = "${bindir}/X Y"
FOO_remove = "${bindir}/X"
the expected result is "Y". Currently this doesn't work since the removed
expressions are not expanded first. This patch adjusts things so the
expressions are expanded before being processed for removal.
Also add a test to ensure this case continues to work.
[YOCTO #6624]
(Bitbake rev: 72a1ca4a104ccab73d6abcbd44db9c2636a58572)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to transfer some of the siggen data from the core/cooker into
the worker instances. There was a partial API created for this but
its ugly and its not possible to extend it from the siggen class.
This patch completes the interface/abstraction for the data and
means the class can extend/customise it in any siggen class.
(Bitbake rev: cf2d642052979d236185c5b8ca2c5478c06e62ae)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Unfortunately it seems like the external use of the cooker
enableDataTracking() function broke at some point since the code that
reads it now runs within BBCooker's constructor. Since this now has to
be done early, add a parameter to Tinfoil's constructor to allow
enabling variable history tracking.
Fixes [YOCTO #6676].
(Bitbake rev: a9439b136f55f3f0e80ff053cd3b159da69ba362)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the improved exception handling added in an earlier commit bitbake
now stops when recieving a SIGWINCH. This happens frequently when
disconnecting and reconnecting tmux sessions and bitbake didn't survive.
Restore old behaviour of ignoring interrupted system calls but keep
proper exception handling for other errors.
(Bitbake rev: 418358a595c75f45b8d15160ec42bbe569562d91)
Signed-off-by: Dan McGregor <dan.mcgregor@usask.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We refactor log saving to go through only one code path.
All logs that happened outside the build (i.e. before build
starting) now will be logged to either toaster_ui.log if the
build command ran in interactive mode, or to the build request
errors if the command ran in managed mode.
This enables proper display of error logs in project page.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the output in the python task/function run files is rather
incomplete and effectively useless. This enhances the code to take
advantage of the bitbake's dependency tracking and extend the output to
include dependencies. This makes the files more usable for debugging
purposes. Since this only happens at python function execution time, the
overhead is minimal in the grand scheme of things.
(Bitbake rev: 02667e048c3e632f857c87177c0022eaf5481802)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we get no profiling oversight into either the main bitbake worker
process, or the overall parsing before task execution. This adds in extra
profiling hooks so we can truly capture all parts of bitbake's execution
into the profile data.
To do this we modify the 'magic' value passed to bitbake-worker to trigger
the profiling, before the configuration data is sent over to the worker.
(Bitbake rev: 446e490bf485b712e5cee733dab5805254cdcad0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This stream redirection of stdout is horrible. pstats takes a stream argument
so lets use that instead.
(Bitbake rev: 93d155f4766e27e7b004d13569aa03961fe89e3b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We should only add user/password options if they're specified as in the
fetch case. Patch from Volker Vogelhuber <v.vogelhuber@digitalendoscopy.de>
(Bitbake rev: 303e6256947f4df4f283b75b7ccfdffa72864d67)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if you have hard dependencies between setscene tasks (like avahi on
base-passwd through useradd.bbclass), other dependencies may not be installed
even if these exist in sstate. For example, avahi -> expat -> pigz-native
(and avahi -> base-passwd) yet if you cleansstate base-passwd:
bitbake gzip-native:do_clean avahi:do_clean expat:do_clean pigz-native:do_clean base-passwd:do_cleansstate
bitbake avahi | tee
you will currently see pigz-native being rebuilt even though it was in
sstate. The fix for this is to continue to iterate dependency chains
around hard blocked dependencies as per this patch.
After this patch is applied, you will see pigz-native installed from sstate.
(Bitbake rev: f787957a224e8c2682a19e5c4a4d9c86bdce52ba)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 587b144ee409d444494d8d7f2d1c53ede8f7c953)
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fixes an issue when trying to use depexp without specifying a
recipe. Before change:
$ bitbake -g -udepexp
Traceback (most recent call last):
File ".../bitbake/bin/bitbake", line 382, in <module>
ret = main()
File ".../bitbake/bin/bitbake", line 370, in main
return ui_module.main(server_connection.connection, server_connection.events, configParams)
File ".../bitbake/lib/bb/ui/depexp.py", line 201, in main
logger.error(cmdline['msg'])
NameError: global name 'logger' is not defined
After change:
$ bitbake -g -udepexp
Please specify a package name for dependency graph generation.
(Bitbake rev: 984ad90b2f1e29634dc79803a4a0404ab0534039)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
since python-2.5 string.endswith() takes a tuple
(Bitbake rev: 86a67a1fd4244da9343dbf14deed1ad0d3003f32)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With a shared work task like gcc, the task can be run from a variety of
different recipes which may have different virtual extensions in place.
Depending on whether gcc-runtime or nativesdk-gcc-runtime's do_preconfigure
task is called for example will change the sorting of the task hashes due
to the way clean_basename currently works.
The correct thing to do here is sort on the base filename first, then any
extension when ordering the hashes. This means we do account for things
like recipes with both a native and non-native dependency but we also fix
the shared work case where we don't care whether it was a virtual version
or not.
(Bitbake rev: 2e80b5d10a5037ed6f0bc227a1f9b42529c87086)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if an exception occurs in an event handler, the server shuts
down but the UI simply hangs. This happens in two places, firstly waiting
for events and secondly, sending events to a server which no longer exists.
The latter does time out, the former does not. These patches improve
both code sections to check if the main server process is alive and if not,
trigger things to shut down gracefully. This avoids the timeout in the
command sending case too.
This resolves various cases where the UI would simply hang indefintely.
(Bitbake rev: ac418e1112ff5f9c3157569316902f7a27fba4b4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Our usage of multitprocessing is problematic. In particular, there is a bug
in python 2.7 multiprocessing where signals are not handled until command
completion instead of immediately.
This adds a workaround into our wrapper function to deal with the issue.
(Bitbake rev: a16185e602b39b71475aa7e9ee80ad2b1f28d0f7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we trigger a shutdown whilst parsing for whatever reason, in some
cases we were not closing down the parser threads. This change
ensures we do so. The function names are not entirely intuitive
but the behaviour is more correct (and commented). The previous bug
with the stdout failure would trigger this one, if there was a cold
cache and parsing was required (but not otherwise).
(Bitbake rev: 25bfa2478f1c3a8eb695e1e5760e06db5be8f2fc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, IOErrors are just passed over due to the broken Exception
clause. A command like "bitbake X | <invalid command>" would break stdout
triggering a traceback. With these changes we print the exceptions, shut down
the server gracefully and exit which is a much nicer behaviour and is less
confusion to the user.
(Bitbake rev: 9544108f7b413038d871ce6ca88232de2f2434d9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if an exception occurs, we just run the idle handler again and
again, usually looping indefintely. Chances are the exception that occurred
will keep occuring and this is not a good place to be.
This was breaking the autobuilders with gigabytes of logs.
At least improve things so the cooker shuts down gracefully when this happens.
Some trace of the original problem may still be present on the console too!
(Bitbake rev: 1f28d8d3311262427938180435b68f0a35c2b330)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie
Christopher Larson
Alex Franco
Benjamin Esquivel
Elliot Smith
Paul Eggleton
Randy Witt
Alexandru DAMIAN
Mariano Lopez
Ed Bartosh
Alexander Kanavin
Andre McCurdy
Aníbal Limón
Aníbal Limón
Olof Johansson
Robert Yang
Leonardo Sandoval
Ross Burton
Felipe F. Tonello
Michael Wood
Ash Charles
Mike Looijmans
Marcin Smoczyński
Liam R. Howlett
Daniel McGregor
Jason Wessel
Anders Darander
Christopher Larson
Rob Woolley
Clemens Lang
Maxin B. John
Pascal Bach
Andy Voltz
Richard Tollerton
Otavio Salvador
Brendan Le Foll
Stefan Müller-Klieser
Gary Thomas
Fredrik Svensson
Oscar Utbult
Koen Kooi
Alejandro Hernandez
Roxana
Chad Nelson
Dan McGregor
Robert P. J. Day
Bernhard Reutner-Fischer