There are some use cases where we want to read a variable but also
set the variable to the value read, effectively locking in any
expansion of it. This adds such a command.
(Bitbake rev: 0c0c524691e3d2ffd9953a106fcc06262cbde910)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Target objects are created before the build if build is
started from UI in build mode. However, in analysis mode Target
objects don't exist and need to be created using information
from bitbake events.
Added new API call get_or_create_targets to retrive existing
target objects or create them if they don't exist yet.
(Bitbake rev: ef69be31d133696bde54605f5a18da660099734c)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for the Subversion fetcher to checkout modules to a custom path
than the module name to avoid checkout is always module - svn is path
based and tag/branch-checkout might break builds because of invaid path specs.
(Bitbake rev: af88f538e61afa1b115be4d7afe00d8477f61750)
Signed-off-by: Jens Rehsack <sno@netbsd.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Additional config files passed to bitbake server with --read and
--postread options are rewritten by client bitbake even if
it doesn't use those options.
This is a show stopper for toaster as toaster command line
builds are based on the assumption that server is aware of
toster configs, provided by --postread option.
This behaviour is fixed by preserving values of --read and
--postread options when bitbake server starts and restoring
them if client bitbake doesn't explicitly specify them.
(Bitbake rev: 02c64f7487ca8ec5d32c440f5002c4b8f64b76a6)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
errno.ENOENT checked if deletion of the fullmirror fails.
Exception was thrown since module not imported
(Bitbake rev: d92ebfc34b69ad5df2d151e6b8299fbb5afa3e5f)
Signed-off-by: Logan Buchy <logan.buchy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BuildStarted event not fully represents build tasks for
the targets. If -c option is used to specify default task
it's not included into the event.
Made build targets to always look as <target>:do_<task>.
Consider default task (do_build or specified by -c command
line option) when normalizing.
(Bitbake rev: 0b0e214e6f53c97ad3d48f622c7fc0ca149956f6)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Critical errors (where a build failed for reasons of
misconfiguration, such as a machine being specified which is not
in a project's layers) were being ignored (only log records
up to ERROR level were being logged to Toaster's db). This meant that
the build would fail but would not correctly report why.
Add support for CRITICAL error levels to the LogMessage model,
include errors at this level in the errors property for a build,
and show errors at this level in the build dashboard.
[YOCTO #8320]
(Bitbake rev: b6eacbca9cacb607de864ab7d093deb296da8226)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we're building using toaster as just a listener to bitbake
(analysis mode) we need to handle the case where the toaster configuration data
isn't present so we don't need to try and update the existing information.
(Bitbake rev: a22faae2c3a5948356ce3cbc73c34509de65d370)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there are more packages listed as installed than we know about from
bitbake, and therefore have insufficient information to be able to
create a Toaster Package object then skip it. Also handle the case where
a dependency references such a package.
Also clarify the error logging.
(Bitbake rev: b4ce793685f70cab3f28cb4329aaaf3878cd62e8)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make sure we associate build data with the built recipe rather than
toaster's configuration copy of the recipe.
(Bitbake rev: 34d4ef7289d72d151ad0acdccab8b99c8c31221e)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Changed logging levels to more appropriate ones.
(Bitbake rev: 27d0360d13af0c698bf3a224b3f0d415f17bb678)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Added ReachableStamps event to the list of known events to
ignore. This should stop toaster throwing error message:
ERROR: Unknown event: <bb.event.ReachableStamps>
(Bitbake rev: cd4137e13af6964858640b78aa7fe6f1612be251)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you run bitbake-diffsigs against two differing sigdata files from
nostamp tasks it shows no difference despite the differing checksum.
Change the code so this shows up as a nostamp 'taint' and at least
makes the issue clearer to the end user.
(Bitbake rev: 97679d18955dadaa34f9450564e44da99984d140)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This firstly prints debug messages which show how bitbake decided to resolve
the virtual/xxx providers which is useful for debugging.
If the siggen has a tasks_resolved() method, it calls this, passing in
the mappings, allowing that to do things with the resolved names.
(Bitbake rev: d473fc84acddfd69a7207affcd89f65ea2ecf730)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When building an execution task graph, bitbake does resolve virtual/xxx
namespaces into specific providers. This data isn't exported anywhere
however.
This adds a function so that runqueue can at least retrieve this data
which can then be used by the system.
(Bitbake rev: ce51a51482d0900060512b24503714a730d72266)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BBPKGS is a confusing name from before we tried to straighten out our
terminology. Its also a mostly unknown variable that isn't in wide use.
I've been asked about it recently and before people start relying more
heavily on it, I'd like to rename it BBTARGETS which better describes
what it does. Its not currently in the manuals, I'd prefer to document
it under the better name. I've not provided any migration path for the
variable since I believe its unused currently.
It allows the targets to built to be specified from a conf file in
addition to those on the commandline.
(Bitbake rev: f60c6a2172bceeb5682dcb738a02c4bf26176566)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prevent edit_bblayers_conf() from adding layer(s) multiple times. This
happened when BBLAYERS variable was "listed" multiple times in
bblayer.conf - i.e. the configuration was split into multiple separate
assignments.
[YOCTO #8316]
(Bitbake rev: 5e423237f9f4ff7e7e03bf066b0142ba4bd82219)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This delete followed up the foreign keys and deleted things that were
not expected to be deleted.
This reverts commit 08000eb27e.
(Bitbake rev: 46b119eb62a5a612fe4c0847862d34f408e556f7)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create a copy of the built layer and the recipes associated with it.
This is so that the user can view the historical information about a
build. i.e. a snapshot of the layer version and artifacts produced at
that build.
(Bitbake rev: 0683d9a2b15e3234a94437aaebac84bfcca1420b)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously this layer relationship was done by trying to match path
information that came back to the buildinfohelper with trying to query for
data in toaster's layers table. This rarely matched due to the lose coupling.
(Bitbake rev: 838e77c7c3c4006abd1327343a004590ab652de9)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Otherwise the logger gets multiple handers (and the user get duplicate
logging output) if another tinfoil instance is initialized after one is
shut down().
(Bitbake rev: 74d67be7a4b591fab2278f7c184f282d11620c62)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The files-in-image.txt file is produced by bitbake after an
image is created, listing all the files in the image.
However, this list doesn't include the root directory ('/').
buildinfohelper.py then tries to construct the filesystem
tree from this file, assuming that every directory apart from
the root directory (which is special-cased) can be assigned
a parent. But because the root directory isn't listed in
files-in-image.txt, an object for the root directory is never
created.
The direct subdirectories of the root ('./bin', './usr' etc.)
then can't be assigned a parent directory, as the object
representing the root directory doesn't exist. This
results in a Target_File lookup error and causes the
directory listing page to fail.
Fix this by creating a fake entry for the root directory
in the Target_File table, so that the direct subdirectories
of / can be assigned a parent. Note that it doesn't matter
that the root is faked, as its properties are never shown
in the directory structure tree.
[YOCTO #8280]
(Bitbake rev: a4015768183e5a3fa39a6c2b4dea0088ca182d80)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Due to re-creating Target objects from bitbake events task information
stored in original objects is lost.
There is no valid reason to remove existing objects. It's safer to query
them instead of re-creating as original object contain more information
than events coming from bitbake.
(Bitbake rev: aab4aff75eefb31aa53885d7735feee5daa294aa)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently this module is dereferencing errno.ENOENT but the python module "errno"
is not imported, which causes a crash when fetching from a git repository.
(Bitbake rev: 93e4c9bb2393b1074f5a01e7eaaac742a59d8086)
Signed-off-by: Romain Perier <romain.perier@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The dependencies of OVERRIDES were not including DEFAULTTUNE in OE-Core.
This is pulled in by a bb.utils.contains() reference which the override
dependency tracking code wasn't accounting for.
This patch ensures we do track contains references too.
(Bitbake rev: f3ee534cb0560dbb5f88a0ffe01e9305bae102e1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Right now, OVERRIDES dependency variables set using ??=, e.g. TARGET_ARCH
in OE-Core don't have their dependencies tracked. This is a bug, fix it.
(Bitbake rev: 944734503768f9e9223ef041f2d7873455418a54)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sadly its not enough to consider the dependencies of OVERRIDES, we
need to resolve their dependencies and so on recursively. If we don't
do this, some variable can be changed and the resulting data store is
incorrect.
(Bitbake rev: 82143ac064d391300e762ba7520ef1f8df18b574)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the values that make up OVERRIDES are themselves overridden,
we end up into some horrible circular logic. Unfortunately some
metadata does depend on this functionality.
e.g:
DEFAULTTUNE_virtclass-multilib-xxx = Y
which changes TUNE_ARCH
which changes TARGET_ARCH
which changes OVERRIDES
As a solution, we iterate override expansion until the values don't
change. If we iterate more than 5 times we abort and tell the user to
report the issue.
(Bitbake rev: 10279697c701e01bf6fdd5e9f92792ef5134807b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were maintaining state in the form of ud.repochanged to determine whether
we need to write the tarball in the case where the tarball already exists, but
this state is maintained only in memory. If we need to update the git repo,
set ud.repochanged to True, and then are interrupted or killed, the tarball
will then be out of date.
Rather than maintaining this state, simply remove the out of date tarball when
we update the git repo, and it will be re-created with updated content in
build_mirror_data.
[YOCTO #6366]
(Bitbake rev: eaaa81393f181432c8586b17ade623f42c9fed2e)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of reproducessing the same line over and over and over, we remove the
current line from the mirror list. This permits us to re-evaluate the list
while excluding all matches that have previousily occured.
Without this fix, adding this test results in a failure:
RuntimeError: maximum recursion depth exceeded in cmp
(Bitbake rev: 24a8e9a5b0ba145ae589178d74365c986ebca325)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The BB_CONSOLELOG variable changes by the time we read it in
BuildInfoHelper. This means that the log file location we
are using is incorrect, so the links to the cooker logs don't
work.
Instead, read it at the point when the BuildStarted event occurs
in toasterui. The BB_CONSOLELOG variable has the correct value
here, so pass that to BuildInfoHelper.
[YOCTO #8209]
(Bitbake rev: 20609eebee0d2318806cf81913e7ce6dc1005507)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
No code changes, just refactoring to allow for functionality
changes by moving things to a separate function.
(Bitbake rev: 2eb934814179ccf42e3d424dabe26b17d013a7ed)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there is a space in a directory name containing a file in file-checksums
(e.g. from a file:// url), you currently get tracebacks from bitbake. This
improves the code to handle colons and spaces in the file-checksums names
since it possible to figure out the correct names.
[YOCTO #8267]
(Bitbake rev: 87282b283921a58426f24fb21151db457c5bca66)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a new bb.utils.signal_on_parent_exit() function so that a process
can register to recieve a signal when the parent dies. There is no
POSIX standard for this and the implementation is Linux specific.
Alternatives would be having an open pipe or polling os.getppid()
for changes but this seems more effective and less invasive to most
of bitbake's code structure.
We need to be able to determine when parents die to ensure child
processes stop running in a variety of circumstances to avoid
locks being held and ensure clean shutdown.
Roughly based on https://gist.github.com/evansd/2346614
(Bitbake rev: 34974f5e30e9b09c016481e4c81c156a5f379784)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if you send a SIGTERM to the bitbake UI process, the system basically
hangs if tasks are executing. This is because the server process doesn't
actually try any kind of shutdown before exiting.
This patch trys executing a stateForceShutdown command first, which is
enough to stop any active tasks before the system exits.
I also noticed that terminate can execute multiple times, once at SIGTERM
from the handler and once from the real exit. Double execution leads to
stack traces and potential hangs (writes to dead pipes), so ensure
the code only can run once.
With these fixes, bitbake much more correctly deals with SIGTERM to the
UI process.
(Bitbake rev: 1032ddddbe3241da02ebb3608a1c40f9123b9e80)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When our clone exists, but is out of date, and the attempt to update it fails,
we don't necessarily want to remove the entire clone, particularly if it's
a large repository.
(Bitbake rev: 19af272ba5256653edeff6acbceeb09e3e478d61)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the main fetch method doesn't support checksums, the user will not be
defining them in the recipe, so we don't want to check them for
premirrors/mirrors either. This ensures that we never error due to missing
checksums on a git mirror tarball.
(Bitbake rev: 24c79bbed361b37f12d3351af13602e3d4386f4c)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using an absolute file URI, there's no host, and the path starts with
'/', the dir under ${DL_DIR}/git2/ ends up starting with '.', so is hidden.
Remove any leading '.' to fix this.
(Bitbake rev: 8dce6964d56b36a77fb113f2ad496cc992a5ff36)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Show the user only the portion of the traceback which was from the metadata,
nothing from bitbake's internal calls.
(Bitbake rev: c45054aef03393fa0bf70e853ddcfc55988493cf)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This gives us the needed context of the original ExpansionError, which is
invaluable when we have a chain of function calls in the expansion.
(Bitbake rev: c514b6fbea77ede1b7871b89592a33ed39b1d71c)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The output of "bitbake, -c listtasks pkg" lists tasks with their real names
(starting with "do_"), but then "bitbake -c do_task" fails, as "do_" always
gets unconditionally prepended to task names. This patch handles this error
by checking whether a task starts with "do_" prior to prepending it with it
when the task runlist is being constructed (and a few other corner cases).
[YOCTO #7818]
(Bitbake rev: dd3050ceef37ac556546e940aa596ce96ef6c8df)
Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you start and suspend a bitbake execution so the bitbake lock is held,
then try and run "bitbake -w '' X", you will see bitbake return an error exit
code but print no message about what happened at all.
The reason is that the -w option creates a "UI" which swallows the messages. The
code which handles this exit failure mode thinks a UI has printed the messages
and therefore doesn't do so.
This adds in an extra parameter to the UI registration code so that we
can figure out whether its a primary UI or not and base decisions on whether
to display information on that instead. This fixes the error shown above and
some bizarre failures on the Yocto Project Autobuilder.
[YOCTO #8239]
(Bitbake rev: d1d60a68c2de40c2984d5040d14251c1be121b0b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake utils' movefile is now prone to malform the destination
file with duplicated file name strings. Fixing it to force a file
name append iff the dest argument is a dir not a file name
(Bitbake rev: 38dd27f7191da002a16c561be3790ce487045b01)
Signed-off-by: Benjamin Esquivel <benjamin.esquivel@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Command line builds are associated with a "default project"
(as we currently require a build to have a project). This
acts as a container for builds initiated outside Toaster.
Currently, this project is marked as the default by its ID
being 0. However, this doesn't work with MySQL, as MySQL
won't allow 0 in a foreign key which references an
autoincrement field.
Instead, use an is_default field to track the default Project
for builds initiated outside Toaster.
Add a method to fetch this default project, rather than fetching
a project with a magic ID.
Add this default project in a migration, rather than as a side
effect of a get_or_create() style method.
Also ensure that builds always have a project explicitly assigned
to avoid any magic with a build's project foreign key defaulting to
0 (as it no longer does).
[YOCTO #7932]
(Bitbake rev: 71b709a1bbc26d89d61873763b467d21e625b274)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When moving a file via the os.rename function, it was missing the
destination file name which caused an OSError
[YOCTO#8180]
(Bitbake rev: b147ba0341d87e077bd2b09ef4355976ecd2d26b)
Signed-off-by: Benjamin Esquivel <benjamin.esquivel@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We want addtask to be able to bring back a deleted task, but we don't want its
previous dependencies to come back with it, so rather than marking a task as
deleted and then skipping tasks marked as such, actually delete the task and
its dependency information in deltask.
While we're in that part of the code, also fix a couple 'not foo in bar'
instances to 'foo not in bar', which is preferred in python.
(Bitbake rev: 94b3f3d6bdfbfa47f7eb3c3de64940a145b2ddd1)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow any listeners for this event (such as buildhistory.bbclass in
OpenEmbedded) to find out if the build was interrupted rather than
completing normally. The value will be 0 if not interrupted, 1 if
interrupted waiting for remaining tasks to complete, or 2 if force
interrupted (stopping any running tasks immediately).
(Bitbake rev: df2b778efd2ecc48f6c5a3ed446f6459f2250035)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're importing the server and UI modules in order to check they are
valid, but it turns out that that has some nasty side-effects. We don't
actually need to do this except when --help is passed or the module
doesn't exist, so rearrange the code so that we only do the module
listing in those two cases.
Additionally, let's just go ahead and catch all errors on import; we
really don't care to have them cause a failure now.
(Bitbake rev: c9dc6d9c86e8b887821a6d00346bd0b09e1da97c)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This exception was triggered in toaster from recent changes and is completely
breaking the whole of bitbake. Add the exception to the list so at least only
toaster is affected.
(Bitbake rev: f64def7cb6069dc1134fcd546bb59e4030c7376f)
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch fixes pylint issues in the toaster build data logger,
toasterui. The following types of warnings are touched here:
* fixing imports
* unused variables are set to _
* logger calls now use lazy evaluation instead of formatting
the string
* correct whitespace identation
* removes unneeded "pass" statements, and extra parantheses
* disable specific pylint warnings when decideing to override
them
(Bitbake rev: 947d47f15048baa967f88e03d80014e88ce152aa)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prompted by issues discovered during running pylint on
the toaster sources, this patch fixes:
* missing import in toaster ui
* improper call of function in toaster ui (logger.debug)
* improper function definitions in bbcontroller
* invalid references to objects in bldcontrol.models
* proper initialization of object fields in ToasterTables
Also inhibiting specific pylint errors in files where
the problems are mis-identified.
(Bitbake rev: 1c71955c416fb68455f7f70669aba4202c411807)
Signed-off-by: Alexandru Damian <alexandru.damian@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake-layers needs to map recipe and class files to the layer they
came from within the show-recipes and show-overlayed commands. However,
it turns out that mapping a file to the layer it came from is not as
trivial as it might seem. To do it properly we need to match the path to
an entry in BBFILES then map that to the collection name using
BBFILE_PATTERN, then map that to the actual layer using variable
history. If it doesn't match any entry in BBFILES, then we can fall back
to BBFILE_PATTERN (to handle classes and conf files).
This fixes the layer name not showing up properly in the output of the
show-recipes and show-overlayed commands for recipes in layers such as
meta-intel that have subdirectories in BBFILE_PATTERN. It also fixes the
priority not showing up in show-layers for such layers.
As part of this I've added a function to VariableHistory which for a
space-separated list variable gives you a dict mapping the items added
to the files in which they were added. I've also fixed
bb.utils.get_file_layer() and reduced some of the duplication by using
this function in bitbake-layers. Also fixes the priority not showing up
for layers such as meta-intel
Fixes [YOCTO #8160].
(Bitbake rev: e852f6cabd7489585477ab567a1afeb2252377ac)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Provide us with a means of showing the list of UIs / server choices for
the command line help, and do the processing in one place for both.
(Bitbake rev: 24035c1daad5a904c3372d21d44191ee8182338f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a file no longer exists, drop it from the cache silently instead of
generating a traceback. This was visible in some cases when a recipe was
deleted when bitbake was resident in memory.
(Bitbake rev: fe105b9042bdac4afd9f38fcf92bfdc2c04ec23f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now we have the bbappends list and all users have been converted over to
use it, we don't need this anymore.
(Bitbake rev: 279770c42d4c63aa2cebce331b55a92a564b50ac)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst collecting this list on the fly may be quicker, doing so within a
function that's meant to *query* the list of bbappends is poor practice.
(Bitbake rev: 5c12aa3b0010d7d1733e54a0ca7d0af465454210)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we can't connect to the server we should error out, because it might
not be that the server is actually dead - it might just be unable to
execute commands.
(Bitbake rev: d4b921676859d6ba4e1922fa4682ee941652f483)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The prefile/postfile options weren't working in memory resident mode
because they weren't being passed through to the server, so ensure that
they do get passed through and that the server is reset when the values
come through.
(Bitbake rev: a3f7dc042fc7b1c308bfd248431930eb8ba50326)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a means of ensuring that synchronous commands that read the results
of the configuration trigger a reparse of the configuration if any
underlying files have changed.
(Bitbake rev: aaf3cc024315450c1674819edf2a4e5cdf293f35)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Unfortunately we were acting on inotify notifications about any files
changing within the watched directories, not just the ones we actually
care about. In OE the build directory is in BBPATH and hence it gets
watched, and we write things like bitbake.lock and
bitbake-cookerdaemon.log to that directory, hence effectively
notifications were being tripped on every bitbake invocation. To avoid
this, record which file/subdirectory we're interested in against each
watched directory so we can ignore any events for files/subdirectories
we don't care about.
Additionally, if we move up to the parent dir, ensure we haven't already
seen it before adding a watch on it (we were previously calling
watcher.add_watch() on the same directory multiple times before in a
typical OE configuration).
(Bitbake rev: bc39b8da34c046b629c43fd0a8cac2efbf1c060f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If sys.exit() is called within a command run over XMLRPC, the XMLRPC
server is effectively trashed (apparently listening but no longer able
to respond to commands). We need to intercept the SystemExit exception
and deal with it as we would any other exception.
(Bitbake rev: 95e391acbc3b4efd6c77637a1ce815012ae0f09b)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With python you should not assign a list as the default value of a
function parameter - because a list is mutable, the result will be that
the first time a value is passed it will actually modify the default.
Reference:
http://docs.python-guide.org/en/latest/writing/gotchas/#mutable-default-arguments
(Bitbake rev: 7859f7388f2e3f675d0e1527cfde18625f36f637)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently bitbake is the only one that knows the relation
between PN and PROVIDES. In some cases it is needed to have
this relation in the data store (the bootloader it's a good
case).
This adds the PROVIDES to the taskdata, so it would be in
the data store as one field of BB_TASKDEPDATA.
(Bitbake rev: a660787311d2961c66c0443bf0e2e094c9baef1b)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you load memory resident bitbake, then change bblayers.conf or other
configuration files, then execute your first command, bitbake will not
notice the changes. This is because it adds the file watches during
inital parsing, which happens at the time the first command is run.
To fix this, we move the addition of the file watches to earlier in
the process, so bitbake then does track them correctly. This also
avoids some issues Paul was seeing with tinfoil2.
(Bitbake rev: b9375c73e736003e66575969c9ea244403e47aeb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've had a long-standing bug where a legitimate error reading a file (IOError
or OSError) is always suppressed as though it was a 'file not found' case. As
a concrete example, if you do a `chmod 000 conf/local.conf`, it'll silently
not parse local.conf, rather than erroring to let the user know about the
problem.
Fix this by handling the ENOENT case specifically.
(Bitbake rev: e691312a3add222b04e7b2f52f8df6abcb9068bf)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tracebacks are of extremely limited usefulness in this context. The exceptions
carry the necessary context already, and the user doesn't care about the calls
in bitbake internals that led to an expansion or parse failure.
(Bitbake rev: 9b95fa94eaae452ac7814f1e67c2f7a6314c52f1)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Set logging level for the Toaster logger to get messages
to toaster_ui.log
Reverted previous workaround.
[YOCTO: #7965]
(Bitbake rev: f1d8e0ff7233f17cff3a56db10d2443b392b407d)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
[\.-_] means "any character between . and _"
What was meant here is certainly "any character from the three characters .-_"
(Bitbake rev: af13eaba627f199f91c048c435b9dbe19c79527f)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the PR server or indeed any other child process takes some time to
exit (which it sometimes does when saving its database), it can end up
holding bitbake.lock after the UI exits, which led to errors if you ran
bitbake commands successively - we saw this when running the PR server
oe-selftest tests in OE-Core. The recent attempt to fix this wasn't
quite right and ended up breaking memory resident bitbake. This time we
close the lock file when cooker shuts down (inside the UI process)
instead of unlocking it, and this is done in the cooker code rather than
the actual UI code so it doesn't matter which UI is in use. Additionally
we report that we're waiting for the lock to be released, using lsof or
fuser if available to list the processes with the lock open.
The 'magic' in the locking is due to all spawned subprocesses of bitbake
holding an open file descriptor to the bitbake.lock. It is automatically
unlocked when all those fds close the file (as all the processes terminate).
We close the UI copy of the lock explicitly, then close the server process
copy, any remaining open copy is therefore some proess exiting.
(The reproducer for the problem is to set PRSERV_HOST = "localhost:0"
and add a call to time.sleep(20) after self.server_close() in
lib/prserv/serv.py, then run "bitbake -p; bitbake -p" ).
Cleanup work done by Paul Eggleton <paul.eggleton@linux.intel.com>.
This reverts bitbake commit 69ecd15aece54753154950c55d7af42f85ad8606 and
e97a9f1528d77503b5c93e48e3de9933fbb9f3cd.
(Bitbake rev: a29780bd43f74b7326fe788dbd65177b86806fcf)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since shell scripts run with 'set -e' the final exit at the end of the
script can only be returning 0. However, for correctness and to follow
the original intention of the 'cleanup' commands, let's fix the typo
and return the success of the last shell function rather than the
success of unhooking the exit trap handler.
(Bitbake rev: bef724057f1ea81571dd3ac5a9cf862f818434b5)
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Calling record() for each override alteration is slow. Since we now expand
overrides dynamically we don't have to record the log data at each alteration,
we can instead print it directly from the existing data stores at variable
history print time using the exact same data stores.
This massively improves performance of the data store when parsing
with bitbake -e for example, it will improve memory overhead as well.
The only downside is that VariableHistory has to poke into the datastore
for some of its data but that seems an acceptable tradeoff rather than
double caching.
(Bitbake rev: 100b447a161ef20fa559e39516cd32fa78e38262)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As things stand now, bitbake -e (which turns on all the caller tracking)
of OE-Core generates around 9.5 million stat calls which is slow and the
largest single thing on the profile data.
This is because infer_caller_details() calls traceback.extract_stack()
which adds line contents to the traceback. This in turn calls python's
internal linecache code which calls stat on every file for every callback.
We don't even use that info. We only even want a single frame of the stack.
Instead, open code for the pieces of information we need. Also, only
obtain the stack once for both halves of the infer_caller_details()
code.
This reduces the number of stat calls to around 0.5 million and significantly
improves parsing with bitbake -e.
(Bitbake rev: 7be76d8f79ea92fd4bd36146eb9a4b86551b526d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We should use the d.xxxVar syntax rather than the older function style.
Also replace 0/1 with the more pythonic True/False.
No functionality changes.
(Bitbake rev: 90fdd69cca951f8bd2ff634f3b42fccd4fc03095)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Minor formatting improvement in generated shell scripts.
Add a newline after the last export emitted by emit_func. Also, remove
chance of putting an extra newline in the middle of the exports, since
we never want to do that.
(Bitbake rev: 5117d9b5d32d5d81adf70fa3e3feac9cef654240)
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
One drawback to tinfoil is that once a cooker is created, it will hold
the cooker lock and stop any other bitbake execution against a directory.
Add a shutdown method to tinfoil, allowing other users to use
the build directory after the tinfoil usage is no longer needed.
(Bitbake rev: e44ce85fe551677fc0dcc1da4f789a0c13093ff1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
One data store changing a variable poked through into a different data
store. This test case replicates that issue where the value 'B' would
become unset/disappear.
We also enhance parsehelper to generate files with an optional suffix
such as bbclass.
(Bitbake rev: 5c4179f58a4e04f1c354df5f17d1860eb403f0ac)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using deepcopy() caused a major performance regression. Turns
out we can work with a shallow copy as long as we force the
recreation of any list that we change, effectively a poor mans
CoW. This isn't too invasive and avoids the huge overhead of
deepcopy so this seems like the best way to have performance and
a working data store.
(Bitbake rev: 32dff4749c32f32e947c42c86f8357b8b607354b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Change the test name to be more specific on what is tested.
(Bitbake rev: 3e39156bc330c4c726058a5b9c13d33e2daad89f)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now latest_versionstring method returns (version, revision) for comply
the new return convention needed by SCM's like git get the current
revision.
bb/tests/fetch.py: Updated wget latest_versionstring test for comply new
convention.
[YOCTO #7605]
(Bitbake rev: 8d454646cbe1b04758ca178d8c6fcfd02b818b7b)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to know the revision associated with the upstream version in
SCM like git, this change broke return convention, oe-core recipeutils
get_recipe_upstream_version need to be updated.
tests/fetch.py: Updated git latest_versionstring test for comply new
convention.
[YOCTO #7605]
(Bitbake rev: fd175dc90024c503134c11cbd83e77d88c406ac8)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Don't limit the tag search for only tags end with ^{}.
(Bitbake rev: 7006ab313766344cf33481228465082ed5977d28)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
dbus sets RDEPENDS_dbus-ptest_class-target = "X". nativesdk as
BBCLASSEXTEND was expanding and setting various RDEPENDS variables
which were clearing overrides queued against RDEPENDS. The problem
was that this was leaking into other data contexts such as the
target case. This was because overridedata was a shallow copy.
Replacing the shallow copy with a deep copy avoids this problem of
leakage between the data stores at a small performance penalty.
(Bitbake rev: 83e2923cd3c8f2e25987132f85b06ce72bc941d7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The recent change to connect through the shell logging functions had an
unexpected side-effect - bb.error() and bb.fatal() cause a flag to be
set internally such that BitBake's UI will not print the full task log
on failure; unfortunately we have in places within the OpenEmbedded
metadata called these shell logging functions under error situations
where we still want to see the full log (i.e., the message we're sending
doesn't include the full error). Thus, provide a mechanism to fatally
exit with an error but unset the flag, utilising the built-in python
logging functionality that allows extra values to be passed in the log
record.
(Bitbake rev: e561b997c55e8537d82aa1339adfff4505cc38b7)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're calling select() to find the fds that have data available, so we
really ought to check the return to see if the extra file is in there
before trying to read from it. This is part of the fix for the
performance regression that this code introduced (5-10 minutes extra in
a reasonable size OE build); the rest is down to an issue in the way
that pseudo currently handles FIFOs and will need to be addressed there,
see: https://bugzilla.yoctoproject.org/show_bug.cgi?id=7993
Solution suggested by Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 3e2db8d7eaa0f14813213d1c95d3ee9efd97dc9d)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a variety of tests which were found to be useful when working
on the data store recently.
(Bitbake rev: 5c5f8da509f6bbc1fad263462142519ef3d54a35)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that overrides get expanded 'on the fly', change appendVar
and prependVar to work using _append and _prepend, else we'd have
to re-implement pieces of getVar and the timing of expansions
becomes problematic.
Using _append/_prepend equivalence gives the behaviour users likley
expect from these functions.
(Bitbake rev: 40d661aaf7a563c6447b073310c5f2fdae6ca3d0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The expand cache can be cleared in the middle of getVar now
due to the use of operations like delVar. We therefore need
to check if variables are in the cache before accessing in
case it was cleared.
(Bitbake rev: a97ce216cfe44136f742383542954bfce027831e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
d.keys() was not returning items which had no base content but the
variable came into existance through overrides. We have to process
self.overridedata to find these other variables.
(Bitbake rev: 77fc57c610d7f65f7e328ad555f525b2f8bdf1d7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This performs better than continually regeneratiing the set().
Also only use set comparisions when its necessary to save some overhead
as issubset() is slower in the single item case.
(Bitbake rev: db9a4eedcf78969ec50bf0e4a8defba8ff0daa4a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When expanding OVERRIDES, its possible someone might try and override a variable
that is used in OVERRIDES. This could lead to infinite recursion. Add in
guards against this.
(Bitbake rev: 07d773369f571028c2cf82dd1f65d9731af6d00e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Updating the value of OVERRIDES whenever it changes turns out to be
extremely expensve/pointless. Instead, clear its value and re-establish
the value when we're going to use it.
This gives significant speed back.
(Bitbake rev: 41cf8d0c92d2d8a33fdad0921e424a0024914be1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using an internal flag for override mapping turns out to be slower
than is optimal, not least as we don't want the keys list to list
variables that have no value other than a potential override expansion.
Maintinaing a seperate cache is therefore more optimal from a speed
perspective.
(Bitbake rev: 1a39ccf66b30b54e1274669cb0e1dc8fc72dba49)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Trying to look up a variable called 'copy' in COW is problematic due
to internal implmentation details, at least avoid tracebacks from this.
Also don't duplicate override history (which is an atrefact of changed
override behaviour) as otherwise the bitbake -e output is convoluted.
(Bitbake rev: dddff5b7b8e6c18515b43389cef35503468b843d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When parsing we find problems if we clear prepends/appends when
setting variables during the initial parsing phases. Later, we actively
want to do this (in what would be post finalisation previously).
To handle this, pass a parsing flag to the operations to control
the correct behaviour for the context.
(Bitbake rev: ae87f5b8bf16191b3201cfb445062938eab992a0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Compeltely remove the replaces functionality and move all overrides
handling to getVar time. We can move the cookie cache into a hidden
flag within the variables themselves.
This removes the need for any of the internal_finalize steps.
This obsolete the need for the _seen_overrides COW cache.
(Bitbake rev: 2a0b73110bede91777ada54d1d89b45fb6034b6c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than repeatedly expanding the value of OVERRIDES, cache
the value and update it when things change.
There were also some bugs introduced in the replaces functionality
which this approach fixes by ensuring the replaces data is updated
at the correct points.
(Bitbake rev: f3b7c3e054ce230ea5c2db5813390383e8dfd6db)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These write operations should clear the expand cache since they can
influence returned variable values but currently do not. Fix this.
(Bitbake rev: a075332c9e13be35f1ae84adc6b29e9137a487ff)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than the heavy lifting in internal_finalize, move the bulk of
the functionality to getVar and rely on a new internal replaces variable
to keep track of any mappings that are needed. This removes the need
for the COW _special_values cache.
This change in functionality also implies we need to track any changes
not only to OVERRIDES but also any variable which OVERIDES depends upon.
Add code to handle this.
Explicitly list FILE as a value OVERRIDES depends upon since it doesn't
automatically get detected and is of key importance to OVERRIDES,
otherwise PN doesn't update when FILE changes.
(Bitbake rev: a6f1377ce3386d274882072d1ae6da3b1834149b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move the update_data functionality into internal data store operations
so the main finalize (update_data) call is a nop.
To make this work we need to call the internal finalization function
whenever OVERRIDES is changed to ensure values get updated correctly.
This has performance issues but the subsequant patches look into this.
(Bitbake rev: 546d9932d6c4413824319a9d780c0d77d2725f4a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since Python 2.7.9 ssl cert validation is enabled by default
see PEP-0476, this causes verification errors on some https
servers so disable by default.
(Bitbake rev: e177170200ece76b36e3f7d5597651fdef67736f)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Only use ProxyHandler opener when exists proxies in env.
(Bitbake rev: 0f062c5d99e12ce20c0e46a8e602448032144ff6)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2/__init__.py: Add connection_cache param in Fetch __init__.
In order to pass connection cache object to checkstatus method.
[YOCTO #7796]
(Bitbake rev: 9fa6407e6cefe66c77467419a8040d6957a6bb01)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Set BatchMode=yes instead of PasswordAuthentication=no. This will make
sftp fail immediately, not only when SSH requires interactive
authentication, but also on errors related to host key verifcation.
(Bitbake rev: 31305853a177735cc9c4553ea8905cd0acfcb100)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Create a branch and named as upstream branch when checkout source
* Set the branch to track remote branch.
(Bitbake rev: 1ba20e4fe9c884515b200589fe379ad5eeda10bd)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you have a situation where you are getting a "Nothing PROVIDES" error
(for example when something you request to build DEPENDS on something
that has been skipped or doesn't exist) it would be useful to be able to
use bitbake -g or bitbake -e to debug it, but currently both of those
are blocked by the error.
This patch adds an "allowincomplete" option to taskdata and uses this
for the -e/-g bitbake options. The NoProvider errors are still
printed and bitbake does return an error exist code but the environment
and task graph files are generated.
[YOCTO #7623]
(Bitbake rev: 2ca36a9f088438a8d1db44119c704f9480b04298)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create a fifo under ${T} for each running task that can be used by the
task to send events back to BitBake. The purpose of this is to allow
OpenEmbedded's logging.bbclass (which provides shell function
equivalents for bb.warn(), bb.note() etc.) to have those functions
trigger the appropriate events within BitBake so that messages are shown
through the BitBake UI rather than just in the task log; to do that we
just call the python functions.
Part of the fix for [YOCTO #5275].
(Bitbake rev: bae330a1f8a59a912eca71177b3c6ba7c92a2f38)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to pass connection cache object to checkstatus function
add fetch parameter.
(Bitbake rev: fbb9c6f5538084e125b58118a86968908e6f895b)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
FetchConnectionCache class acts as a container for socket connections
useful when implement connection cache into fetcher modules.
(Bitbake rev: 454da2cd17539ceb9caad6d76f034757e44ee12f)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before exiting the UI, unlocks the bitbake.lock owned by cooker; this
way consecutive bitbake executions can lock it again without trouble.
[Yocto #7941]
(Bitbake rev: 69ecd15aece54753154950c55d7af42f85ad8606)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Function which calls cooker's unlock method, which in turn unlocks bitbake.lock
file.
(Bitbake rev: e97a9f1528d77503b5c93e48e3de9933fbb9f3cd)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, once set, DATE and TIME in the metadata remain unchanged.
This is suboptimal for cases where cooker is reused. This ties setting
the variables into the BuildStarted event which seems like a more
appropriate time to do so.
It also changes BUILDNAME to be based off DATE/TIME by default if not
already set so that the data is more consistent. We therefore need to
expand the value rather than the previous default of not doing so.
This change does mean the date/time values are in sync across all
variables too.
It does mean bitbake now has special knowledge of DATE/TIME but that
would seen unavoidable (other than doing this in event handlers which
has its own set of downsides).
[YOCTO #5187]
(Bitbake rev: f883cf240266ee7be2cbd8971a8164cf4df9e372)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The incredibly useful recipe:do_task syntax on the command line isn't documented
at all. This isn't much but it's better than nothing.
(Bitbake rev: 7f4c07886ecff4ac77fdd2165bedd179099fcf19)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is nessary when specified branch with submodules is different then
default (master) branch.
[YOCTO #7771]
(Bitbake rev: f7b0b5e33e00f3ce0744322eee93835ee76bf184)
Signed-off-by: Felipe F. Tonello <eu@felipetonello.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make sure the error value is initialised and update the string match
that we have to identify bitbake's current startup status.
Patch contribution from Eduard Bartosh
(Bitbake rev: 915ba08a8a3013e9787e564f2ffd8698c948f433)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These aid when debugging parts of the codeparser cache since the
object contents becomes identifiable.
(Bitbake rev: 344b098c7eafc2bcc5c6b44ea47985bc0cb446b5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The main parser and other code copes with empty python functions but
the python codeparser does not. Fix this to avoid errors with code like:
python do_build () {
}
which currently results in the obtuse:
"Failure expanding variable do_build: IndexError: string index out of range"
[YOCTO #7829]
(Bitbake rev: e4f594c670189e04d58ce7d160fc1d86123620af)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
* do_unpack error:
abort: repository DL_DIR/hg/vim.googlecode.com/hg/vim not found!
* The mirror tarball doesn't work
- Add the build_mirror_data to create the tarball
- Unpack the mirror tarball when needed
* The hg files will put in the dir like git: DL_DIR/hg, it was
DL_DIR/hg/path/to/src/uri/path in the past.
(Bitbake rev: 378647281ae883dd726f1e1b775dd35ef6a0e8d1)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fetching directories with p4 worked but single files did not. This
patch from Helmut Auer (helmut.auer@harman.com) fixes that issue.
[YOCTO #7891]
(Bitbake rev: 39da6579901c62a83ed9319c2016c58fbbc108fe)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Events can call each other recursively, e.g. an event handler can call
bb.note which in turn generates another event. If these loop, it
can lead to multiple deletions of 'd' from __builtins__ which
can fail since __builtins__ is global scope.
Add handling to only remove 'd' when we added it and it wasn't already
present.
(Bitbake rev: b45952650ce8f470f124df36185b79e0d3a1783a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some handlers hook on BuildComplete so it avoids certain event races
to finish the command after the BuildComplete event is sent out.
This means the UI is available to handle events until the command
completes.
What appears to be a race on one of the sanity tests for event handlers
triggered this change although the failure is hard to reproduce.
[YOCTO #7921]
(Bitbake rev: e42d7c47a06fbb5981e0313478c8e3656b99f4e7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch verifies that BRBE is set before trying to use
it to read the checkout paths. This is needed for builds
ran outside Toaster control.
(Bitbake rev: e04807cd3135c9de96cc7f79245f329c24618b85)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We remove the "timespent", "errors_no" and "warnings_no" fields
in favor of computing the needed values at runtime. This prevents
inconsistencies in the UI.
Also removeing all references to BuildRequests from the interface -
all build details now display in the build dashboard.
Minor fixes related to data logging.
(Bitbake rev: 44f37394ed3e4ca02f940be172fe4395b0ee0f7d)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a set of fixes that repair the interface after
we switched from displaying BuildRequest data to Build data
in the formerly "managed" mode.
(Bitbake rev: 57f790b0c56297af8c83d5def8461bd5d61fe4af)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds logic to complete changing the interface
from showing BuildRequests to showing Build data.
The BuildRequest data is now transformed in Build data with
proper Toaster exceptions being recorded instead of listing
problems during startup as build errors.
(Bitbake rev: 51a41172d0b390370f9a38696b1ac65666ada4d2)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake does really weird things with negative or zero numbers of threads
which is confusing to the user. Add a sanity check for this.
When you have code doing arithmetic on the values and a VM reconfigures
to only a single thread, negative numbers are easier than you'd think.
(Bitbake rev: 32166ac3c85ff3c04081580ae76bd63590d6ff3e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we see a BBHandledException in runqueue, the understanding is the system
handled it, printing a log and traceback is just confusing.
Therefore only print these in the cases where its an unknown/unhandled
exception.
(Bitbake rev: 29d28e22ce431c3d3aabdb88ff4d8cca67a1cfad)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we see a BBHandledException in the idle handler, the understanding
is the system handled it, printing a log and traceback is just confusing.
Therefore only print these in the cases where its an unknown/unhandled
exception.
(Bitbake rev: d88ecc2bc44dce8fd92ca3a2c0fd4124a5e464fa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if the fakeworker failes to start the output from bitbake is confusing.
Improve the error handling to give a clear indication of what failed.
Patch from Chris Larson.
(Bitbake rev: ad286d6fed7a580bec36a92c7b7e205322ac407b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To quote Chris Larson:
"""
e.data.getVar() gets a bit old in a large event handler, and it means a
simple handler has to be modified if switching between an event handler
(e.g. RecipeParsed) and anonymous python. I think it would make sense
to restore the 'd' convention here to align with python elsewhere.
It'd just be a convenience, d==e.data, to avoid the common pattern of
setting it at the top of the event handler.
"""
I couldn't find a way to inject 'd' via locals/globals due to the use
of a function parameter so this left __builtins__ as the only way
I could find to make this work.
[YOCTO #7668]
(Bitbake rev: 44ac81e5281fb62ad00e2f79a9d754118ea62526)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than just use d.getVar(X), use the more explict d.getVar(X, False)
since at some point in the future, having the default of expansion would
be nice. This is the first step towards that.
This patch was mostly made using the command:
sed -e 's:\(getVar([^,()]*\)\s*):\1, False):g' -i `grep -ril getVar *`
(Bitbake rev: 659ef95c9b8aced3c4ded81c48bcc0fbde4d429f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The metadata can potentially use such an event to clean up any
"unreachable" data, solving several problems we currently have
where obsolete data may continue to exist in the shared areas.
(Bitbake rev: c5e6f929f3d5eeb7954660dea62611c58b795ff8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we're not going to change the value of FILE, or we know it isn't
going to have changed (ext == bbclass), don't set FILE.
This avoids messy looking history of the variable as well as optimises
parsing speed slightly.
(Bitbake rev: 88e4600aa66dda2e6c807f9d97af8982bcd8817b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake logger now sets a default project if the TOASTER_BRBE
or TOASTER_PROJECT Bitbake variables are not set.
This a necessary step in getting all builds under a project,
as to unify the MANAGED and interactive modes.
Other small fixes are included, related to the size of the
fields in the database.
(Bitbake rev: 5e0bf388f4e5c1cc493ac8264785e631bad2f672)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This refactoring brings the "local_path" of the
layer from the Layer object to the Layer_Version object, which
is more appropriate as different checkouts of the same
Layer may live in different directories.
This enables us to store Recipe file paths relative to a Layer_Version
at all times, aleviating the need to store full file paths in the
database. We also turn the prefix of the path (e.g. virtual:native
path name space) into a pathflag field.
In turn, this solves the problem of mis-identification of tasks based
on the recipe file paths, since we can also match the namespace of the
file paths on the recipe files.
[YOCTO #7594]
(Bitbake rev: ec43dc569e370767c709dec225cbee0c99151c19)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
At least in theory, the order the keys are expanded in can make
a difference, particularly if there is key overlap.
We also want to ensure that any underlying base key is processed
before any overridden version of that variable (FOO before FOO_x)
which helps the update_data removal code I've been testing.
(Bitbake rev: 863b6add24c211d64ba7931647084321f2d65889)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding a dependency when errors occur accessing a file when calling
handle() is not the correct thing to do. THe handle() code calls
resolve_file() which can raise an exception without ever touching
"fn" itself, it has also already marked all the dependencies correctly.
This leads to bogus files being resolved to the local cwd and
hence triggers reparses for no good reason.
The solution is to simply remove the bogus dependency.
(Bitbake rev: 366af3be1cffd64e4a79c15990c1e05869022c14)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst bitbake has done this for a long time, the behaviour of resolving
class files against cwd is not desirable. This can be seen during
base configuration parsing when looking for base.bbclass where a dependency
on cwd is added. If cwd then changes, the cache is invalid and triggers a
re-parse.
The only real option is to drop this entry and if files can't be found, we
fix BBPATH in the cases where it needs fixing. I didn't find any in the
random selection of layers I tested parsing locally.
(Bitbake rev: 508aad9d5db7e51328b1fd6ee53b4bc3720a30b7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the cooker parsing processes each dump an individual profile
which is ok, but means absolute numbers of function calls for a given load
can be tricky to determine as parsing of recipes may go to different pool
threads on different runs.
This change collects up the individual thread parsing results and processes
them into one profile output. The profile processing function in utils
needed tweaks to allow this to work.
(Bitbake rev: d3d2541aacd1ea560da0d8b25a3ea3f0563dee70)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This ensures basic key expansion works and that overlapping
keys generate a log message.
(Bitbake rev: ed5a8954ac923eda9750a636c5bb5b95ffce664f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows us to write tests which ensure a particular action generates
a particular log message.
(Bitbake rev: b30ee0aba51a35a194a4338b988f93ece1ed281c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was incorrect, it *must* be done next to the rename as a previous
may overlap with the current one and we need to detect this case.
I'll add a test case to better catch this problem in future.
(Bitbake rev: 5e9d0911cd86e980ab310cc0d79c9383fbc2c844)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This comparison is interesting even in the case of empty vales. Enabling
this warning actually found a bug in the metadata in avahi. Make the
code handle None specifically and also remove the dead code path in the
second if statement.
(Bitbake rev: a4cd4c56284812efb2a2bc0c8667ddad073f8e94)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By doing this we can take advantage of the expansion cache before
starting write operations on the data store.
(Bitbake rev: 702b42a47904f2378dd819e7463b3206883c2651)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the number of task dependencies change you currently get
a traceback when using diffsigs.
(Bitbake rev: c6798b431571aae18bb8699ac6e3ec75b731d719)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Like the Git fetcher, the Mercurial fetcher shouldn't expect recipes to
provide a checksum. As described [1], recipes using a mercurial
fetcher that don't provide a checksum will fail in a the repository has
previously been downloaded and archived.
Credit to Rafaël Carré for figuring out the bug.
[1] https://www.mail-archive.com/openembedded-devel@lists.openembedded.org/msg41328.html
(Bitbake rev: 2df35a25b4968f64adfa673d5b73442c1a30829d)
Signed-off-by: Ash Charles <ashcharles@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
gitpkgv_revision returns a sortable revision number that can be used
in the PKGV variable for example. To mimic meta-openembedded gitpkgv
behaviour to provide a sortable revision numner, one could set the
following:
PKGV = "1.0+${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"
This would yield a package version like "1.0+69+fb5eb80".
(Bitbake rev: 989c08f62aff7b707c25c692c23284f16506b7bc)
Signed-off-by: Mike Looijmans <mike.looijmans@topic.nl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The idea here is to support package version numbering similar to gitpkgv in
meta-openembedded. This commit is the first step towards such functionality.
The original plan was to add a "get_pretty_srcrev" method to the fetcher, as
per Richard's suggestion [1]. While writing this, I noticed that it would
become a copy of get_srcrev with only two lines changed. So to create something
more Pythonic than a boolean argument and conditionals around the calls to the
fetcher's sortable_revision, I just made the method to be called on the fetcher
an argument to the method. Defaulting to 'sortable_revision' prevents affecting
existing code.
Now if the git fetcher were to implement, say 'gitpkgv_revision' one could
set the following in a recipe:
PKGV="1.2+${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"
and this would yield the same result as gitpkgv's GITPKGV variable.
See for the discussion leading to this change:
[1] http://lists.openembedded.org/pipermail/openembedded-core/2015-January/100345.html
(Bitbake rev: 2f1f4483493cc290f5d2c07f9906e90eaea2f4c1)
Signed-off-by: Mike Looijmans <mike.looijmans@topic.nl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Pass proper repository url without arguments after a semicolon.
Executing checkuri on a rule with git repository in SRC_URI does
not report errors when working offline because wrong repository
url is passed to the ls-remote command. For example
"bitbake -c checkuri glibc" command executes:
"git -c core.fsyncobjectfiles=0 ls-remote git://sourceware.org/git/glibc.git;branch=release/2.21/master"
command in a shell subprocess to determine if url is valid.
Shell subprocess executes in fact 2 commands:
"git -c core.fsyncobjectfiles=0 ls-remote git://sourceware.org/git/glibc.git"
and
"branch=release/2.21/master"
First one returns 127 or 128 depending on error but second one
returns 0 because it is just env variable setup. Therefore we're not catching
connection error.
[YOCTO #7558]
(Bitbake rev: efa44d04137977f883db4a643b0f774e91514722)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix several bugs and add some useful enhancements to make this into a
more generic metadata editing function:
* Support modifying function values (name must be specified ending with
"()")
* Support dropping values by returning None as the new value
* Split out edit_metadata() function to provide same functionality
on a list/iterable
* Pass operation to callback and allow function to return them
* Pass current output lines to callback so they can be modified
* Fix handling of single-quoted values
* Handle :=, =+, .=, and =. operators
* Support arbitrary indent string
* Support indenting by length of assignment (by specifying -1)
* Fix typo in variablename - intentspc -> indentspc
* Expand function docstring to cover arguments / usage
* Add a parameter to enable matching names with overrides applied
* Add some bitbake-selftest tests
Note that this does change the expected signature of the callback
function. The only known caller is in lib/bb/utils.py itself; I doubt
anyone else has made extensive use of this function yet.
(Bitbake rev: 20059e4d5ab9bf0f32c781ccb208da3c95818018)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Fix unchanged assignments being dropped if other lines changed
* Fix not passing variable name from single-line assignments to the
function
* Fix not trimming the trailing quote from values
(Bitbake rev: 0b0c82f49cf2de887967d305768cbd95314bb171)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In certain contexts it can be useful to find the layer that a file (e.g.
a recipe) appears in.
Implements [YOCTO #7723].
(Bitbake rev: 3bf9c8830c5d5eea5502230d5af84ebd87ad5849)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add some tests of mirrors or mirrors to the fetcher unittests.
(Bitbake rev: e33d82bc10283d533f928836d56a6f0af80ea5c1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
First, when building mirror urls we don't do any fetching so we should never
be calling clean functions.
Currently, if a mirror url fails, we don't process it further to see
if there are any mirrors of the mirror.
We should do this even when the mirror url fails, else we may miss out
on valid/useful mappings, particularly in the case of file:// urls.
(Bitbake rev: b7fd3ec9994f664b17fc86423e6e7afac07e897b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow GIT_SMART_HTTP to be passed through to the fetch command so that
servers that cannot use GIT_SMART_HTTP can be used by the fetcher.
(Bitbake rev: e5c97a85bed0436d48eeaac2e32962cfb5371d2f)
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The URL is not sent when _latest_revision generates and exception.
When performing the sanity checks it is not possible to know the URI that failed.
This add the URL when latest_revision generates an exception.
[YOCTO: #7592]
(Bitbake rev: 9f2115b07a55cb14e4a74dc6fbd3707c28a234d0)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add some sanity checks on the parsing state engine when returning data
so that incomplete functions raise parse errors.
This means a recipe doing:
do_somefunction {
echo 1
VAR = "1"
will now raise a ParseError. To get the right file/line information,
__infunc__ was changed to a list.
[YOCTO #7633]
(Bitbake rev: 6b54a72638f57882d4fd5aab96b2752a09e065af)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fetcher calls os.chdir() in a number of places, which can affect
other tests (since the directory it changes into gets deleted) - let's
just put the current directory back to where it was when we're done.
(This fixes bb.tests.Path.test_unsafe_delete_path failing if it was run
as part of a full bitbake-selftest run, where the fetcher tests get to
run before it.)
(Bitbake rev: b1653855c74f86909c9f329ed6d2b10391c28395)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you're interested in using the checked out repository for development
(e.g. in OE with devtool) then you ideally want the origin remote to
point to the repository it was fetched from, so just set that after
cloning.
(As part of this I did a minor refactor so we have one function to
generate the repository URL, which was already in two places.)
Fixes [YOCTO #7756].
(Bitbake rev: 80ecd1c54d4c748cee3a7ce0d64013a346e7671e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases we need to check specifically for siginfo files, in
some cases we need to check for the actual sstate objects themselves.
Therefore make this a parameter to the function. A fallback to the
previous function style is maintained for now.
(Bitbake rev: 18d3a03e1b07c98b2dce46eb94f30de1a2b4320b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BB_ALLOWED_NETWORKS is a list of hosts that the fetcher will be allowed
to use when BB_NO_NETWORK is not set.
If BB_NO_NETWORK is set, then networking is still disabled.
If BB_ALLOWED_NETWORKS is not set, the behaviour remains the same as
today.
If BB_NO_NETWORK is NOT set, and BB_ALLOWED_NETWORKS is configured, then
only the hosts in the list are usable by the fetcher.
eg:
BB_ALLOWED_NETWORKS="yoctoproject.org git.gnu.org"
The fetcher will be able to download from yoctoproject.org, git.gnu.org,
but not ftp.gnu.org or any other hostname that is not in the list.
There is also limited support for wildcards on the beginning of the
hosts, so BB_ALLOWED_NETWORKS="*.gnu.org" with match git.gnu.org and
ftp.gnu.org as well as foo.git.gnu.org
(Bitbake rev: c7263096ba31ba45daeeb9de90c1cb9ebef24a28)
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use hg clone and hg pull to copy the source into the build
directory rather than taring up the cloned repository and
untarring in the destination.
This allows submodules to be cloned. While here, make the default
behaviour keep the hg scm data to match the behaviour of the git
fetcher.
(Bitbake rev: f002b1ca80cb542a4ed0c06c53c914cd5e076565)
Signed-off-by: Daniel McGregor <daniel.mcgregor@vecima.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Setting BB_ENV_EXTRAWHITE to include GIT_SSL_CAINFO is enough to
get "git ls-remote" to work for recipe parsing but it is not enough
to get the fetcher to clone properly.
This is because the fetcher has its own idea about what variables
should be exported in the environment for some operations.
It is desirable to use alternate CA Bundles for internal testing prior
to using public keys for https, so we should allow the GIT_SSL_CAINFO
to pass through.
(Bitbake rev: 40ff92282bbf32cf644b021bf7cbb1f393dbb856)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Moving the code to close the build and instantiate a fresh
buildinfohelper object to BuildCompleted event, as the
CommandCompleted/Failed/Exit events come in too early.
(Bitbake rev: af63abe88327fd5c1b3d7c00a84d9e408ef23285)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch modifies the toasterui to properly return the exit
code based on the errors found in the toaster itself.
The upload event file API call will not delete event logs for which
toasterui showed an error. This will facilitate debugging.
Minor enhancement in the buildinfohelper to reduce the number
of lookups on unknown layer objects (prevented testing of the patch).
(Bitbake rev: 1ddd6a9e4280a4adf971132ff1fe7ec9b3252905)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Introduced build mtime cache structure. Reset it before the build
to prevent bitbake from crashing when build/tmp/stamps hierarchy
is removed.
[YOCTO: #7562]
(Bitbake rev: f8590547a198a78334debdf14bf40acb50c22ecc)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake program and core versions must match.
Moved __version__ from main.py back to bin/bitbake.
Implemented check for version match in bin/bitbake.
(Bitbake rev: 2fe7d8c574ddf6a30278cff1a5a5c4089dc56d6d)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
tbs
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some error messages were lost because BBMainException was
inherited from bb.BBHandledException. When bb.BBHandledException
is processed error messages are not printed as they suppose to
be printed before raising this exception.
Stopped to inherit BBMainException from bb.BBHandledException.
Handled BBMainException in bin/bitbake and printed error message
to the stderr.
(Bitbake rev: c8e2a40c4e9865ebef9936d23644f2602a5c90f5)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add some very basic safeguard against recursively deleting paths such
as / and /home in the event of bugs or user mistakes.
Addresses [YOCTO #7620].
(Bitbake rev: 56cddeb9e1e4d249f84ccd6ef65db245636e38ea)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Implemented processing of ~ in bblayer's paths if HOME
environment variable is approved.
(Bitbake rev: 3b8a656d3ccb543c32696229184ebf12237ad38e)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the subpath parameter to the git fetcher ends with a trailing '/',
bb.utils.prunedir() will be called on '/'...
Fixes [YOCTO #7620].
(Bitbake rev: 380a3fb372c8b0a53dd7528562e6e7a222dc76ef)
Signed-off-by: Anders Darander <anders@chargestorm.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This ensures that _append, _prepend, overrides, etc are functional when used
on sstate variables (e.g. SSTATE_DIR).
[YOCTO #7564]
(Bitbake rev: 2e683c25b856b431198573f7f352d841587275e6)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
self.appendlist is a dict and as such unordered. This can lead to cases
where appends with different names (e.g. x_%.bbappend vs. x_123.bbappend)
can be reordered in application which in turn reorders the variables
that those bbappend files might touch. Reorderd variables changes the sstate
cache signatures causing real world issues.
To avoid this, use a list for the append files instead.
This patch is conservative and just adds a new data structure alongside
the existing one and uses it to resolve the core issue. Later patches
(post release) can handle some of the wider but less problematic ones
(e.g. issues in bitbake-layers flatten).
[YOCTO #7511]
(Bitbake rev: f980f060cd0d1e7fe5011f3c325c1b254f05eccf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since toasterui acts as the user-facing UI, we need
to run sanity checks in order to let the GUI display proper
warnings and stop the build if something is wrong.
(Bitbake rev: 260dd77fa771ae3b777134f4178d344e96b6f3d6)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch prevents tracebacks and instead logs exceptions
that may happen during event processing.
[YOCTO #7216]
(Bitbake rev: 0412631fb4a15ff42bf5ee46a77920fa558ae358)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Moved most of functionality of bin/bitbake to lib/bb/main.py
to be able to call bitbake from python code.
(Bitbake rev: d377f7f88d73f4e5d2dffef03d6acee809827ac6)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch fixes the recipe and layer identification by path
when Toaster uses relative paths.
(Bitbake rev: a92bb33a3ceacab2bfee9df1c39a202832866970)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We improve logging and signalling of errors in the
toaster_ui.log to facilitate debugging on remote systems.
(Bitbake rev: 3cd248f99b90367bd41aab81e255fc1912434890)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It used 5 spaces as the indent.
(Bitbake rev: 162d35ed53d34b28b153adf643044e7f105fcff1)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We modify the toasterui to log relative recipe paths
in order to maintain consistency with data fetched from
the layer sources.
(Bitbake rev: 253d69e88fd68729196ad43c15e8733527d76198)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In build mode, instead of creating our own layer objects,
we identify the layer objects that the build system set up.
[YOCTO #7378]
(Bitbake rev: 22962b540ace6868cb357c0fd13f01ffd24449c4)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Every time the bitbake show versions command (bitbake -s) is run it creates
a 100k log file.
The consolelogfile is disabled for show environment and disabling show
versions would make the behaviour match.
(Bitbake rev: dee0ba94e39ea49c1e9261a5e8932356e3bb7c57)
Signed-off-by: Rob Woolley <rob.woolley@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Any exceptions that occur in calls to logging methods are automatically
suppressed, including exceptions due to broken pipes.
However, the knotty summary messages are printed directly to stdout, which
means that any broken pipes will cause an exception traceback in python.
By wrapping the summary section in a try / catch block we can check for
IOError exceptions caused by broken pipes and let them pass.
(Bitbake rev: 146e7e157f97b676858ecff583dd53800d997253)
Signed-off-by: Rob Woolley <rob.woolley@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The recent inotify changes are causing a 100% cpu usage issue in the
idle handlers. To avoid this, we update the idle functions to optionally
report a float value which is the delay before the function needs to be
called again. 1 second is fine for the inotify handler, in reality its
more like 0.1s due to the default idle function sleep.
This reverts performance regressions of 1.5 minutes on a kernel build
and ~5-6 minutes on a image from scratch.
(Bitbake rev: 0e0ba408c2dce14a0fabd3fdf61d8465a031495b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
[YOCTO #5571] -- https://bugzilla.yoctoproject.org/show_bug.cgi?id=5571
The following workflow (whether accidentally or deliberately) would
previously not result in a checksum error, but would be helpful to do
so:
- Write a recipe with correct checksums
- Fetch the sources for this recipe using bitbake
- Change the checksums
Since the bitbake fetcher writes a done stamp after the initial download
and does not verify the checksums again (even if they are changed in the
recipe afterwards), the change of checksums is silently ignored.
Fix this without the overhead of computing the checksums from scratch on
every do_fetch by storing them in pickled format in the done stamp and
verifying that they still match those in the recipe.
(Bitbake rev: fbd4a0d422cf7f43db2f9eab2e47c41246a9031e)
Signed-off-by: Clemens Lang <clemens.lang@bmw-carit.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch makes sure to delete an idle function that
raises an exception for the xmlrpc server.
The counterpart functionality in the process server was
added with:
commit db50630948.
duthor: Richard Purdie <richard.purdie@linuxfoundation.org>
Date: Wed Aug 20 22:31:06 2014 +0000
bitbake: process: Deal with infinite looping of the server
This patch fixes
[YOCTO #7316]
(Bitbake rev: e7c9a6788d969c901fd6394416ac3936e62c4c72)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch fixes a bug where if the build is force stopped,
subsequent clients cannot connect to the server due to
unnecessary limits on setFeature.
Additionally, we make sure that the state is properly reset
even if the BuildCompleted event firing excepts for some reason.
(Bitbake rev: 0b66b05169688aa4ddc4c54d175bb961b2f27fec)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We roll back the client connection if some error happens,
like during setFeatures, as to leave the server accessible
to other clients.
(Bitbake rev: 4e4a2ee2f05f8741b2e09263e328420363975b02)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The inotify facility monitoring changes to the config files
could be overwhelmed by massive changes to the watched files
while server is running.
This patch adds verification the notification watches to the
server idle functions, in addition to the cooker updateCache
command which executes only infrequently, thus preventing
overflowing the notification buffer.
[YOCTO #7316]
(Bitbake rev: 996e663fd5c254292f44eca46f5fdc95af897f98)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some databases, notably the SQLite3 adapter, force autocommits even if
the autocommit is turned off. The behavious is tracked in this bug:
http://bugs.python.org/issue8145#msg109965
Django refuses to work with autocommit off in this case, so we have
to take the same precautions when using manual transaction support.
[YOCTO #7363]
[YOCTO #7365]
(Bitbake rev: 90231ab63a129fa344d461c2911898ea0f07f206)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch fixes a problem where set sstate scene tasks
were not identified, causing cache attempt not being recorded.
[YOCTO #7223]
(Bitbake rev: 8a326a9a5a08981f1b7960e02fdb8a9436db16fb)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch disables autocommit for inserting build data,
effectively updating all build data in a single transaction.
This is a purely performance improvement patch, as the transaction
will always be commited.
Similar manual transaction handling in the layer source update
method. Added feedback messages during update method.
[YOCTO #7140]
(Bitbake rev: 3978c819e797f857235499a4b8ec238134f1c028)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch brings in cooker log saving and proper download links.
* toasterui will now write the cooker log file if running in managed
mode
* the BuildRequest has a new state, REQ_ARCHIVE, indicating that the
build is completed, and the artifacts are ready to be grabbed
* the runbuild test execution commands will gather needed artifacts,
and save them to a storage directory selected during Toaster setup.
* the build dashboard, project builds and all builds pages have
permanent links for the cooker log
[YOCTO #7220]
[YOCTO #7206]
(Bitbake rev: fad80e36c9da663b000cdf2cb3c75440c6431d84)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add set of PN in data because now latest_versionstring use it for
validate version directory searching.
(Bitbake rev: 2e4a03db967ac1459b2764108fc54c4566a7e371)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix sustition for rc, beta and alpha releses from -N to N weight.
(Bitbake rev: 63a9e60a6d80cfd2693ec1a6359785dc19f98e1f)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for scan every version directory using _check_latest_version
makes code more robust because sometimes upstream projects publish
new directories without files, causing don't find version.
To support this new behaviour remove _check_latest_dir and replace for
_check_latest_version_by_dir,
(Bitbake rev: 1a75b3707743c32eec9d2cf566fb6bbea9f73784)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add _check_latest_version_by_dir this function provides support
for scan every directory newer than current dir in order to get
latest_versionstring, example:
http://somedoamin.com/project/v2.1/http://somedoamin.com/project/v3.0/
Change return of _vercmp from True/False to -1/0/1 to provide test
when current directory is equal to newer directory this helps to
scan the same directory to get minor versions, example:
http://somedoamin.com/project/v2.1/project-v2.1.2.tgzhttp://somedoamin.com/project/v2.1/project-v2.1.6.tgz
(Bitbake rev: 5f7c5eb218a221165f59a0f4dd48d2d97f756193)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to reduce code duplication now compile package_regex in
_init_regexes instead of make this decision at _check_latest_version,
(Bitbake rev: e7284e3ad0e7dd91ed59dfbf8450ef62e89c7e54)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for get group only if exist in regex, this enables to use
this function in _check_latestversion regardless if the regex is generic
or specified by REGEX_URI.
(Bitbake rev: 1127af5b8c458929c4685b0326f86870ed09442e)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've had versioned dependency support in LAYERDEPENDS for quite a long
time, but I can say with pretty good certainty that almost nobody has
used it up to now because it was too strict - the specified version had
to exactly match the version in your configuration or you would get an
error; there was no "greater than or equal" option, which is usually
what you will want given that LAYERVERSION does get bumped from time to
time.
However, users mismatching layer branches and then having their builds
fail later on with some incomprehensible error is still a pretty common
problem. We can't simply use the git branch because not everyone is
always on a branch and the branch names don't always match up (and
that's not an issue). To provide a practical means to address branch
mismatching, I have reworked LAYERDEPENDS version specifications to use
the more familiar "dependency (>= version)" syntax as used with package
dependencies, support non-integer versions, and clarified the error
message a little. If we then take care to bump the version on every
breaking change, it is at least possible to have layers depend on these
changes when they update to match; we can now even support a major.minor
scheme to allow retrospectively adding a version limiter to old branches
when a new branch is created and yet still allow the old branch minor
version to be bumped if needed.
Fixes [YOCTO #5991].
(Bitbake rev: 408be9cdf2b1e32e64ea488d8051a546fb54c144)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This scheme is used for versioning recipes that are pre-release (alpha,
beta, etc.) within OpenEmbedded, so add some tests to ensure the
appropriate comparison results still hold true.
(Bitbake rev: 3a9eefe27f29a4593d6298f0427ac5f3e9183377)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We really want an error rather than the version to just be silently
skipped when the operator is missing (e.g. "somepackage (1.0)" was
specified instead of "somepackage (>= 1.0)".)
(Bitbake rev: b6dc946f477adc40d68da16e2f2580cb3b4a10db)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake -g -u depexp <package> segfaults when DISPLAY is not set
properly. Fix it with a proper check.
[YOCTO #7299]
(Bitbake rev: f35e9bd7b59c180fe9a3d9177efb57b92d9cd373)
Signed-off-by: Maxin B. John <maxin.john@enea.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch improves the logging facilities for toaster in order
to help diagnose bugs that happen on user machines.
The logs are stored now under "/tmp/toaster_$$" where $$ is a
PID-based unique identifier. On shutdown, toaster will automatically
erase all logs unless errors are listed in the log file.
On error, Toaster provides suggestions on what to do.
This patch includes a minor fix found as a result of logging
improvements.
(Bitbake rev: 8a8248f7b7e30469f592e2f8adbf6ce21e8685c5)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Given these assignments:
TEST="a b c d"
TEST_remove = "b d"
TEST evaluates to "a c". However, if the _remove override is given as a
variable:
TEST="a b c d"
FOO = "b d"
TEST_remove = "${FOO}
TEST evaluates to "a b c d", because when FOO is expanded it isn't split into a
list.
Solve this by splitting all members of removeactive once they've been expanded.
[ YOCTO #7272 ]
(Bitbake rev: 207013b6dde82f9654f9be996695c8335b95a288)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The _remove operator isn't working correctly when used with a variable that
expands to several items, so add a test case to exercise this path.
(Bitbake rev: cb2a62a5fbffb358528a85b46c1fc6383286cb9d)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These only have one instance created which means your subsequent datastores
can contain echos of previous ones. Obviously this is not the behaviour
we want/expect. It doesn't affect bitbake too badly as we only have one
datastore, it does massively potentially break our selftests though.
Thanks to Tim Amsell for pointing out the now obvious problem!
(Bitbake rev: 9facf3604759b00e8fe99f929353d46f8b8ba5cb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
ToasterUI filters build artifacts by extension in order
to determine if a build artifact is an image or not.
Using IMAGE_FSTYPES for this purpose is not correct as
the varible value holding image extensions is just a coincidence.
So we just look if the filename contains the "rootfs" magic
string, which is a pretty good approximation.
[YOCTO #7213]
(Bitbake rev: b11e8bd626e0212ee72914529c3d92d1dd718674)
Signed-off-by: Alexandru Damian <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The get/set_taskdata functions are now part of the API of the class,
ensure they exist in the base class definition so the noop handler
works.
[YOCTO #7233]
(Bitbake rev: 9b5b1bd7d77e3f5886f6c557d3b750de1f6d6025)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Users expect operators like _remove to work on this variable. We need
to use expanded_data to ensure this happens correctly.
[YOCTO #7135]
(Bitbake rev: cc4c8478fc547ea0ebf827a8d319496b39f25684)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
event_data would be better named expanded_data, then we can widen its scope
to other places in cooker where we need to access an expanded data store.
We certainly don't want multiple expanded data stores.
(Bitbake rev: 1a3c1c9203e1a1452314954f1cfd771e5c1ce89b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was a report that bitbake -e | less would use 100% cpu when it shouldn't
really. The issue appears to be a bogus file descriptor in the select call. We
shouldn't be blocking if there is event data pending to a *reader* from server
context.
[YOCTO #7138]
(Bitbake rev: 8f166e1a0f3574ae7d1e917a8bb403b87bad15bf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add a call that saves any queued events to the build
[YOCTO #7021]
(Bitbake rev: 4f5b19d453da64749affc1c27ec51b013bedc71a)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When built package regex for use in searching upstream version in sites,
the package name need to be escaped to avoid usage of special regex
char.
For example when search for gtk+, '+' need to be escaped.
(Bitbake rev: 1aa1de4b0c5bd34466e04844bbc371933736be59)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Trap the ENOSPC error and translate it into a human readable error
message, which is good for debugging.
(Bitbake rev: 2b084dff6ff0d274fbbf7ab07022507f7249e427)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We currently add crazy numbers of watches on files. The per user limit is 8192
by default and on a system handling multiple builds, this can be an issue.
We don't need to watch all files individually, we can watch the directory containing
the file instead. This gives better resource utilisation and better performance
further reverting some of the performance regression seen with the introduction
of pyinotify.
(Bitbake rev: a2d441237916a99405b800c1a3dc39f860100a8c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the environment changes, we need memory resident bitbake to adapt to those
changes. This adds in functionality to handle this alongside the configuration
option handling code. This means that the common usage:
MACHINE=X bitbake Y
now works with the memory resident server.
(Bitbake rev: 4d1343010da757a0c126bc22475354da44aaf8e3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tweak the comments on get_srcrev() to better describe its function.
(Bitbake rev: b4d40f1ac7b32990c456cce261f99a5a157b5ae5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A bug slipped in the toaster ui that prevented saving of
build configuration despite the data being retrieved
from the server. This patch fixes the shaming mistake.
[YOCTO #7117]
(Bitbake rev: 8118f465b9f87c66b2a741008f69198ac5fea901)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The import statements here are plain bizarre. Remove them, tweaking
some of the function calls to match current practices. I can't find any
reason these old imports are as they are.
(Bitbake rev: 4c2f1fe51a13ddc97e518327714292af46b9e1ab)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Looking at this function I had no idea what oldfn was, I doubt anyone
else would either without looking up what the caller does. "parentfn"
would seem a more appropriate name so rename it.
(Bitbake rev: fc70ed596703a1aa954223b169d4ad51193a6ec1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Benchmarks show that the introduction of pyinotify regressed
performance. This patch ensures we only call the add_watch() function
for new entries, not ones we've already processed which does improve
performance as measured by "time bitbake -p".
This doesn't completely remove the overhead but it does substantially
reduce it.
(Bitbake rev: 493361f35f6cc332d4ea359a2695622c2c91a9c2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We try and add watches for files that don't exist but if they did, would influence
the parser. The parent directory of these files may not exist, in which case we need
to watch any parent that does exist for changes. This change implements that fallback
handling.
(Bitbake rev: 979ddbe4b7340d7cf2f432f6b1eba1c58d55ff42)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Memory resident bitbake has one current flaw, changes in the base configuration
are not noticed by bitbake. The parsing cache is also refreshed on each invocation
of bitbake (although the mtime cache is not cleared so its pointless).
This change adds in pyinotify support and adds two different watchers, one
for the base configuration and one for the parsed recipes.
Changes in the latter will trigger a reparse (and an update of the mtime cache).
The former will trigger a complete reload of the configuration.
Note that this code will also correctly handle creation of new configuration files
since the __depends and __base_depends variables already track these for cache
correctness purposes.
We could be a little more clever about parsing cache invalidation, right now we just
invalidate the whole thing and recheck. For now, its better than what we have and doesn't
seem to perform that badly though.
For education and QA purposes I can document a workflow that illustrates this:
$ source oe-init-build-env-memres
$ time bitbake bash
[base configuration is loaded, recipes are parsed, bash builds]
$ time bitbake bash
[command returns quickly since all caches are valid]
$ touch ../meta/classes/gettext.bbclass
$ time bitbake bash
[reparse is triggered, time is longer than above]
$ echo 'FOO = "1"' >> conf/local.conf
$ time bitbake bash
[reparse is triggered, but with a base configuration reload too]
As far as changes go, I like this one a lot, it makes memory resident bitbake
truly usable and may be the tweak we need to make it the default.
The new pyinotify dependency is covered in the previous commit.
(Bitbake rev: 0557d03c170fba8d7efe82be1b9641d0eb229213)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When urls ends with trailing slash os.path.basename return "" [1]
and built urldata.localpath only with DL_DIR, it causes that
donestamp is built as DL_DIR + '.done' and seems that ssh resource
was already download.
[YOCTO #6448]
[1] https://docs.python.org/2/library/os.path.html#os.path.basename
(Bitbake rev: 47992591349bab2c12741b937096e41085399087)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the code ignores lightweight tags which has caused some user
complaints. We can't put the right search list in place easily since
the results don't come back in a good order, head happens to sort
before tags.
In the end I refactored the function so we get the complete list of
remotes and then we can filter it ourselves in the order we chose,
including checking for light weight tags, preferring the proper ones.
Hopefully this resolves the issues people have been seeing.
[YOCTO #6881]
(Bitbake rev: 07ad307065bb15a48f0015b9e4a643201abdc283)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A dash character is illegal in function names in sh (but not bash). Since
our shell tasks run under sh and the shell parser is sh based, EXPORT_FUNCTIONS
won't work with class names containing a dash.
We can't change sh, we can ensure the user is warned about the problem
straight away though.
[YOCTO #7006]
(Bitbake rev: 86704281b79e524dccccc88cbf996b299b33bae2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if you reference a file url, its checksum is included in the
task hash, however if you change to a different file at a different
location, perhaps taking advantage of the FILESPATH functionality, the
system will not reparse the file in question and change its checksum to
match the new file.
To correctly handle this, the system not only needs to know if the
existing file still exists or not, but also check the existance
of every file it would have looked at when computing the original file.
We already do this in the bitbake parsing code for class inclusion. This
change uses the same technique to log the file list we looked at and
if files in these locations exist when they previously did not, to
invalidate and reparse the file.
Since data stored in the cache is flattened text, we have to use a string
form of the data and split on the ":" character which is ugly, but is
an internal detail we can improve later if a better method is found.
The cache version changes to trigger a reparse since the previous
cache data is now incompatible.
[YOCTO #7019]
(Bitbake rev: 6c0706a28d72c591f1b75b6e3f3b645859387c7e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases for cache purpoes we not only need to know which file
is going to be used but also which paths were considered. Add a
localpaths method which includes the history.
The core which() funciton already supports this, this just extends
the function to preserve the extra data we need. localpath becomes
just a special case of the case with history.
(Bitbake rev: ea5efeac5c1f7986666c979f789786f29fc1619a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the cooker is in an error state, we shouldn't continue to try parsing.
This fixes an issue where an invalid PR server is detected when bitbake
is started and ensures bitbake exits cleanly rather than hanging.
[YOCTO #6934]
(Bitbake rev: 294bb9cad294423d4f8998405ceff58655f12660)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a little crude as the usage basically ignores that you're doing
an append operation, but for a lot of cases it will be sufficient.
(Bitbake rev: 24a28205ab680b6cc645d97b76c9855920608229)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Fix add-layer error message when a layer is already in BBLAYERS
* Ensure we show an error message if we can't find BBLAYERS at all
(Bitbake rev: 1c743fd2103730e27699dd55efc6914d3b0c3702)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Hardcoding a temp directory is bad practice and leads to races between
the tests. There is no longer any good reason for doing this, drop it
and ensure the files get cleaned up correctly.
(Bitbake rev: 10a47b1ec7470c9e8c4ffe0bb35cdf6d1bb2ee2e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you copy the datastore, then delete a key, it should not exist in
d.keys(). This adds a test to cover the recently found data store bug.
(Bitbake rev: 16d5f40ad20fd08bf7a4d0e36200c739b5a9f59e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The versionstring tests hit the network so should only run when
network tests are enabled.
Also remove the print statement which confuses the test output and
add it to the test failure message instead.
(Bitbake rev: 6c046660cfc4fb3792a42aeafff91a13f68a2e89)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you copy the datastore, then delete a variable, it still shows up
in d.keys() when it should not. This patch addresses the issue.
(Bitbake rev: f28ee1bb03cb32d3757fbef67c9fbe143e3dadfa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to fix multithread usage of latest_versionstring moves package_custom_regex_comp
from class to method level because need to be defined by package.
Remove code for build url's with /download suffix because it's deprecated since you can
specify the download directory using package_regex.inc file.
(Bitbake rev: 231cae9f9b552ec6737795c098d1de426b5adcbc)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a BBFILE_PATTERN_IGNORE_EMPTY variable to allow ignoring the fact
that a regex specified in BBFILE_PATTERN for a particular collection
doesn't match any recipes. This will be used in OpenEmbedded in the
workspace layers created by "devtool" which may not always contain any
recipes (which is not cause for warning the user).
(Bitbake rev: 19c74aaa2836a88d666f0032452fac521689ab6b)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Add a generic edit_metadata_file() function to modify variable
assignments in any metadata file (conf, bb, bbappend) using a callback
for flexibility
* Add a specific edit_bblayers_conf() function to modify
conf/bblayers.conf and add and/or remove layers from the BBLAYERS
value within it.
(Bitbake rev: aa03a28b442549dd8ffe92ae4d6390f62202a76a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a nostamp task is depended on by a non-nostamp task, then we want the
signature of that task to change such that it re-executes afterwards.
This is an unusual situation, but we want this to work in OE in
externalsrc.bbclass so that compilation happens every time it is
requested.
(Bitbake rev: 73498afc3d45beede5b8f24a9acd523a1663b793)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Running bitbake inside make results in the exported environment variable
MAKEOVERRIDES="${-*-command-variables-*-}", which the shell chokes on
when trying to expand it. But of course, it probably shouldn't have been
trying to expand it in the first place -- so just escape the dollar
sign.
(Bitbake rev: 18cd0ce6a55c9065c3f1bf223b47d817b5efcd8f)
Signed-off-by: Richard Tollerton <rich.tollerton@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds a "-w/--write-log" option to bitbake
that writes an event log file for the current build.
The name of the file is passed as a parameter to the "-w"
argument. If the parameter is the empty string '', the file
name is generated in the form bitbake_eventlog_DATE.json,
where DATE is the current date and time, with second precision.
The "-w" option can also be supplied as the BBEVENTLOG
environment variable.
We add a script, toater-eventreplay, that reads an event
log file and loads the data into a Toaster database, creating
a build entry.
We modify the toasterui to fix minor issues with reading
events from an event log file.
Performance impact is undetectable under no-task executed builds.
(Bitbake rev: 1befb4a783bb7b7b387d4b5ee08830d9516f1ac2)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We update and add logs throughout the code in order to help
with development. The extra logging is turned off by default,
but it can be enabled by using environment variables.
All logging happens through the Python logging facilities.
The toaster UI will save a log of all incoming events if the
TOASTER_EVENTLOG variable is set.
If TOASTER_SQLDEBUG is set all DB queries will be logged.
If TOASTER_DEVEL is set and the django-fresh module is available,
the module is enabled to allow auto-reload of pages when the
source is changed.
(Bitbake rev: 10c27450601b4d24bbb273bd0e053498807d1060)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add a BuildArtifacts class to store data about files
discovered during the build process and not stored anywhere
else.
Small cosmetic changes in the toasterui.
Add model methods to return file path display data relative
to the build environment instead of absolute file paths.
[YOCTO #6834]
(Bitbake rev: bbe24d912869312d561be199b2c029b0c898e049)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After Ctrl+C is pressed to interrupt bitbake, it loops continually, running
at 100% cpu. This patch selects on the correct file descriptors resolving
the excess cpu usage.
(Bitbake rev: 497404e8484b7ca7c11e459bf0845642156eb677)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The shellshock patches changed the way bash functions are exported.
Unfortunately different distros used slightly different formats,
Fedora went with BASH_FUNC_XXX()=() { echo foo; } and Ubuntu went with
BASH_FUNC_foo%%=() { echo foo; }.
The former causes errors in dealing with out output from emit_env,
the functions are not exported in either case any more.
This patch handles things so the functions work as expected in either
case.
[YOCTO #6880]
(Bitbake rev: f28f37220e7787721a31b659521a1c44ebea92bf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you don't explicitly specify to use a global variable when doing an
assignment, you will be setting a local variable instead, which means
this function wasn't working at all. It explains some odd behaviour we
have seen in the layer index where event handlers were sometimes
bleeding into other contexts where they should not have been.
(Bitbake rev: ac45ea848901b0f6cd23087b662dde8ce9cd807e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using external tinfoil-based utilities, it is useful to be able to
turn off most of the event handlers; for example sstate_eventhandler
doesn't like being sent events for any recipe which has been skipped.
(Bitbake rev: 41236c28985a3b66b3737382a94e39dbf6480160)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we're fetching outside of the context of a recipe, it's handy to be
able to disable checksum functionality so you don't get a meaningless
warning about the signatures being missing.
(Bitbake rev: 49dbcfbc56a206964acc5de761bba31be0283ba1)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a function that allows executing a flat python function (defined
with def funcname(args): ...).
(Bitbake rev: 20e6939ebcb62e08a9a7ad586a915dfe368136a0)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The defaultval field is intended to be internal and the only use of that field
outside of data.py is to skip over it when iterating over a value's flags.
For clarity and convenience, rename the field to _defaultval so that it is
considered internal and not exposed through the data API.
(Bitbake rev: 2800958dadaa5c055ba21d52c98d842d360f0785)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if an invalid PR service is selected the server will error
with a traceback. This is because its set into the error state and the
setFeature code will then fail since its not in the initial state.
Modifying the featureset in the error state is acceptable, we just need
to ensure we don't trigger a reset, that would happen from whichever
code handles the error.
[YOCTO #6934]
(Bitbake rev: c52841445d8db8f84c4da34203b195fea5874247)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update test case for cups is needed because match only 2.0.0
versions see VERSION=2\.0\.0 in the previous string.
(Bitbake rev: 148aba30155f4de17f6c6fb9b4c37f08a2db202b)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some upstream sites put the name of the package in the body of href tags,
i.e. <a href="#43">somepackage-v1.4.10.tar.gz </a>.
(Bitbake rev: 4fa8542a8880d5749fbb3382706e011b012024d0)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Validate if package contain version string if not return the current
version cases for spectrum-fw and corpus recipes.
_check_latest_version return the latest version available don't
take into account the current version previous this only return
the upstream version if it greater than the current version.
(Bitbake rev: 91a7ac8c7f87f98e366585cf9720ec35b0790bae)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
package_custom_regex_comp is built with the current package name and
then used to search upstream version this reduces custom regex'es in
sites that have different packages in the same directory.
(Bitbake rev: a69d5d48c4bf9a3df3121cc4e69ba5d7e947ad8d)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Latest version string only try to find latest directory when REGEX_URI
isn't specified to avoid unnecessary processing and makes code easier
(Bitbake rev: afc33ec7cdb7d8ee3602a23fa973551ca5510ac4)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch brings in changes that allow a toasterUI coming in
from 'master' branch to record data from a 'daisy' or 'dizzy'
bitbake server.
This is needed to allow Toaster to record builds running
on older branch releases.
(Bitbake rev: 8d75e28e0688a6520311afce36543175f36910b3)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we have a managed build, we match the layers used for build
with the layers configured for project, as we know where the layers
are coming from
[YOCTO #6962]
(Bitbake rev: e02ec052a62cbc476bdac65cb7cea1167ce04781)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Changing ToasterUI to log toaster exceptions on a different level than
build errors.
Updating the build dashboard to show Toaster exceptions.
We add extra logging to console for exceptions.
Fixed a problem where packages database entries were created instead of
being looked up in the database, conficting with entries created to
satisfy dependency information.
Toaster now checks for invalid states at startup and performs needed
cleanups.
Removed loading reference to jquery-ui.min.css as we do not have this
file.
(Bitbake rev: 2378812bc24d433125fb940f110154f0ce638448)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
1) Run "bitbake recipe" in the terminal
2) Close the terminal while building
3) $ ps aux | grep bitbake-worker
There will be many processes, and they will keep the resources (e.g.,
memory), and won't exit unless kill or kill -9.
(Bitbake rev: 40d2ae0723de2bf5fee343faafb4afda40546839)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The oe.utils.contains function has been removed from OE-Core metadata
as the references for it has been replaced to use the
bb.utils.contains.
(Bitbake rev: 5cfdebe7a67dccc7552ff80c1ccc970e36d562df)
Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This state was renamed in bitbake 12e9d3 but this use of it wasn't changed.
[ YOCTO #5445 ]
(Bitbake rev: 196106100e2dedfacce95be0ee6def94c5a80f27)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
loginfo "op" was being set to invalid values in saveConfigurationVar it was
working because set is a python global but append is not. This replaces both
the "op" with strings and removes the crash when calling appendConfigurationVar
(Bitbake rev: 2a8e847de85546d43600a561f9c63aa36bd69222)
Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Improve the performance of data logging in toasterui.
We modify the data queries used to:
* cache searching in memory
* insert in bulk (i.e. multiple values per insert, where possible)
On development test rig (networked mysql), on no-op build,
time for data recording is reduced from 4:10 to 1:30 (minutes).
We also improve the logging, so it is easier to detect
toasterui errors.
(Bitbake rev: d42784432f927f58730caf80546c66772e0fec89)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Being able to query whether updated versions of a url are available
is useful, not least for the package reporting system. Since such code
is closely linked to the url type and the url itself, the fetcher
makes a locical place to contain this code.
For wget based urls this means taking upstream directory listings
and searching those for later versions, returning those that are
found.
The patch also adds unittests for this function so that if
improvements are made, the original test urls can be used
to evaulate the those changes.
This is based on code from Irina Patru <irina.patru@intel.com>.
(Bitbake rev: a8272e22b7819e0e8afd8e291d276f5f28fc0007)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Being able to generate a version string representing the most recent git commit
given git is useful, not least for the package reporting system.
This adds in a latest_versionstring method to the git fetcher
which allows users to query the latest version using ls-remote
and filtering the responses.
The patch also adds unittests for this function so that if
improvements are made, the original test urls can be used
to evaulate the those changes.
This is based on code from Irina Patru <irina.patru@intel.com>.
(Bitbake rev: f71c8c0354e87fed80bc845db6728e6e18ce9c4d)
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we split variables only at whitespaces, a slipped in tab will render
a value unremovable.
(Bitbake rev: 9f171ea755644ecd9d2b3d7ed13bf8ec09ec917a)
Signed-off-by: Stefan Müller-Klieser <s.mueller-klieser@phytec.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Similar to the last shared work task signature bug, we've found another
one. Looking at the improved output of diffsigs in this case:
runtaskdeps changed from [
'autoconf_2.69.bb.do_populate_sysroot:virtual:native',
'gnu-config_20120814.bb.do_populate_sysroot:virtual:native',
'libgcc-initial_4.9.bb.do_patch:virtual:nativesdk'
] to [
'autoconf_2.69.bb.do_populate_sysroot:virtual:native',
'gcc-crosssdk-initial_4.9.bb.do_patch',
'gnu-config_20120814.bb.do_populate_sysroot:virtual:native'
]
so we can get a different task hash since libgcc sorts before gnu-config
and gcc sorts after it. We could do with a way of fixing this, the best
I can come up with is to include a single parent directory. Since
recipes are never at the top of any metadata trees I've seen, this
should suffice for now.
I'm planning to burn the concept of shared work within bitbake
and do something at the metadata level in the 1.8 timeframe as its just
too fragile as things stand and hard to fix well.
(Bitbake rev: d753644c67d163f338f2bdc3d600203e8b1a5734)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is useful code to double check the computed checksum value if nothing
else. Might as well have it in tree.
(Bitbake rev: 54ecf96c6f031927ee2410f6efde4e16f19bbf66)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Order of runtaskdeps is important. If the hashes differ we should print output.
This is complicated by shared work where the filenames themselves can differ,
but the checksum should not.
This fixes a case where two different checksums could show no output with
bitbake-diffsigs.
(Bitbake rev: 40c95cb9def282dc88234cd72ff462d7a01e47c1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add improvements in data reading, following
issues discovered in testing.
- elapsed_time is now read from buildstats
- we add safeguards to not fail logging if the build was triggered
with a toaster_brbe configuration, but it's running in
1.6 mode
- added log markups for build finish to let other programs
known when the work is done.
[YOCTO #6833]
[YOCTO #6685]
[YOCTO #6887]
(Bitbake rev: 0b225035cefee3d3713a93f9a432e5e4d4e174f1)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This has always bothered me:
NOTE: Preparing runqueue
NOTE: Executing RunQueue Tasks
This patch changes the messages to be consistent.
(Bitbake rev: 72ac9f9227fbfb4dc8b933b357d21aa0e4060959)
Signed-off-by: Gary Thomas <gary@mlbassoc.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This makes it possble to fetch Gerrit review references which are
normally stored under refs/changes.
Please disregard previous patch with the same topic.
(Bitbake rev: 268e9c0c6830e8e621c418f20c2ca12dc840e48b)
Signed-off-by: Fredrik Svensson <fredrik.svensson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We avoid a race between the setting the TOASTER_BRBE variable
and reading the variable in toaster ui by supplying the variable
at server startup time through the toaster.conf post-read file.
Additional small changes are included, including marking the
build request with the environment id of where the build took place.
(Bitbake rev: 7c333350418c4140e6c988c5272940f8057d327d)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We change the saving of the build id to the build request as
soon is the build is created, as to allow for a consistent
display of build data while build is in progress.
(Bitbake rev: 9504ca6a69ba6da21f88b3cc77fa5910d886b6c3)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to consistently use LogMessage.INFO/WARNING/ERROR to make sure toaster knows
how to categories these rather than passing in the "raw" loglevel value
which in best case comes from python logging but worst case any value.
[YOCTO 6885]
(Bitbake rev: 3aa13bc1d8218c97c76581a895fa5f03ff807fbe)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
See http://www.nongnu.org/lzip/lzip.html for details on the compression
format.
(Bitbake rev: 9027b1273b5405c7269b013604ab417771b5eafe)
Signed-off-by: Koen Kooi <koen@dominion.thruhere.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Forcing strict to be a string, to avoid problems when performing comparisons
[YOCTO #6762]
(Bitbake rev: b8ed2098bdea2afd93ab4e3e1b834f3a31cb60de)
Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
hob was adding the redundant characters "\1" in SSTATE_MIRRORS variable. If
needed it is expected the user will add this instead so remove the code
that was doing this.
[YOCTO #6600]
(Bitbake rev: 73bf120062fc00c7e26dc4e77a7d140658d89daf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fixes urls of the form file://some/path/file;subdir=b. It also
adds in a couple of tests so we now tests these corner cases.
(Bitbake rev: 46306912a96444790efa9418d934dfdd36773ba1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Check if the 'subdir' parameter exists and assign it to 'destdir' so that
files are copied in ${WORKDIR}/destdir. This fixes urls that are of the form
file://a;subdir=b.
(Bitbake rev: 836a986b365eb9798563ec08d90b346596de7791)
Signed-off-by: Roxana Ciobanu <roxana.ciobanu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Writing a log that the filesystem isn't being monitored for inode usage just
confuses users who are not aware about the nature of inodes in their filesystem,
so don't say anything, just silently disable the monitor. In general this only
happens on filesystems which don't have a limit on inodes.
(Bitbake rev: ca93bc84ee5fb94a50c11c47e4d212d7da649e24)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I could apply the label "release_1.0" to a super project that contains
many sub projects. If my recipes have SRC_URI's that use that label but
grab different sub-folders, than there's a bug where the cached localfile
(tar.gz) will not be unique and reused at the wrong times.
SRC_URI = "p4://perforce::1234@//depot/SuperProject/MiniProjectAAA/...;label=release_1.0 \
p4://perforce::1234@//depot/SuperProject/MiniProjectBBB/...;label=release_1.0"
(Bitbake rev: 3b5b1703b77490116dda895b29737cea51a3d6a0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using options like -k, -f, -v and so on with the memory resident
server, they'd currently only be set on the initial values passed to
the original command. This ensures they now match those specified
on the commandline for the options where this makes sense.
To make this work, a command to update the options on the server side
is required so this is added.
[YOCTO #5292]
(Bitbake rev: 1c75cc4d0c8b606c1fe76e6bf60bf6a32298b105)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you have:
FOO = "${bindir}/X Y"
FOO_remove = "${bindir}/X"
the expected result is "Y". Currently this doesn't work since the removed
expressions are not expanded first. This patch adjusts things so the
expressions are expanded before being processed for removal.
Also add a test to ensure this case continues to work.
[YOCTO #6624]
(Bitbake rev: 72a1ca4a104ccab73d6abcbd44db9c2636a58572)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to transfer some of the siggen data from the core/cooker into
the worker instances. There was a partial API created for this but
its ugly and its not possible to extend it from the siggen class.
This patch completes the interface/abstraction for the data and
means the class can extend/customise it in any siggen class.
(Bitbake rev: cf2d642052979d236185c5b8ca2c5478c06e62ae)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Unfortunately it seems like the external use of the cooker
enableDataTracking() function broke at some point since the code that
reads it now runs within BBCooker's constructor. Since this now has to
be done early, add a parameter to Tinfoil's constructor to allow
enabling variable history tracking.
Fixes [YOCTO #6676].
(Bitbake rev: a9439b136f55f3f0e80ff053cd3b159da69ba362)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the improved exception handling added in an earlier commit bitbake
now stops when recieving a SIGWINCH. This happens frequently when
disconnecting and reconnecting tmux sessions and bitbake didn't survive.
Restore old behaviour of ignoring interrupted system calls but keep
proper exception handling for other errors.
(Bitbake rev: 418358a595c75f45b8d15160ec42bbe569562d91)
Signed-off-by: Dan McGregor <dan.mcgregor@usask.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We refactor log saving to go through only one code path.
All logs that happened outside the build (i.e. before build
starting) now will be logged to either toaster_ui.log if the
build command ran in interactive mode, or to the build request
errors if the command ran in managed mode.
This enables proper display of error logs in project page.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the output in the python task/function run files is rather
incomplete and effectively useless. This enhances the code to take
advantage of the bitbake's dependency tracking and extend the output to
include dependencies. This makes the files more usable for debugging
purposes. Since this only happens at python function execution time, the
overhead is minimal in the grand scheme of things.
(Bitbake rev: 02667e048c3e632f857c87177c0022eaf5481802)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we get no profiling oversight into either the main bitbake worker
process, or the overall parsing before task execution. This adds in extra
profiling hooks so we can truly capture all parts of bitbake's execution
into the profile data.
To do this we modify the 'magic' value passed to bitbake-worker to trigger
the profiling, before the configuration data is sent over to the worker.
(Bitbake rev: 446e490bf485b712e5cee733dab5805254cdcad0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This stream redirection of stdout is horrible. pstats takes a stream argument
so lets use that instead.
(Bitbake rev: 93d155f4766e27e7b004d13569aa03961fe89e3b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We should only add user/password options if they're specified as in the
fetch case. Patch from Volker Vogelhuber <v.vogelhuber@digitalendoscopy.de>
(Bitbake rev: 303e6256947f4df4f283b75b7ccfdffa72864d67)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if you have hard dependencies between setscene tasks (like avahi on
base-passwd through useradd.bbclass), other dependencies may not be installed
even if these exist in sstate. For example, avahi -> expat -> pigz-native
(and avahi -> base-passwd) yet if you cleansstate base-passwd:
bitbake gzip-native:do_clean avahi:do_clean expat:do_clean pigz-native:do_clean base-passwd:do_cleansstate
bitbake avahi | tee
you will currently see pigz-native being rebuilt even though it was in
sstate. The fix for this is to continue to iterate dependency chains
around hard blocked dependencies as per this patch.
After this patch is applied, you will see pigz-native installed from sstate.
(Bitbake rev: f787957a224e8c2682a19e5c4a4d9c86bdce52ba)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 587b144ee409d444494d8d7f2d1c53ede8f7c953)
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fixes an issue when trying to use depexp without specifying a
recipe. Before change:
$ bitbake -g -udepexp
Traceback (most recent call last):
File ".../bitbake/bin/bitbake", line 382, in <module>
ret = main()
File ".../bitbake/bin/bitbake", line 370, in main
return ui_module.main(server_connection.connection, server_connection.events, configParams)
File ".../bitbake/lib/bb/ui/depexp.py", line 201, in main
logger.error(cmdline['msg'])
NameError: global name 'logger' is not defined
After change:
$ bitbake -g -udepexp
Please specify a package name for dependency graph generation.
(Bitbake rev: 984ad90b2f1e29634dc79803a4a0404ab0534039)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
since python-2.5 string.endswith() takes a tuple
(Bitbake rev: 86a67a1fd4244da9343dbf14deed1ad0d3003f32)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With a shared work task like gcc, the task can be run from a variety of
different recipes which may have different virtual extensions in place.
Depending on whether gcc-runtime or nativesdk-gcc-runtime's do_preconfigure
task is called for example will change the sorting of the task hashes due
to the way clean_basename currently works.
The correct thing to do here is sort on the base filename first, then any
extension when ordering the hashes. This means we do account for things
like recipes with both a native and non-native dependency but we also fix
the shared work case where we don't care whether it was a virtual version
or not.
(Bitbake rev: 2e80b5d10a5037ed6f0bc227a1f9b42529c87086)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if an exception occurs in an event handler, the server shuts
down but the UI simply hangs. This happens in two places, firstly waiting
for events and secondly, sending events to a server which no longer exists.
The latter does time out, the former does not. These patches improve
both code sections to check if the main server process is alive and if not,
trigger things to shut down gracefully. This avoids the timeout in the
command sending case too.
This resolves various cases where the UI would simply hang indefintely.
(Bitbake rev: ac418e1112ff5f9c3157569316902f7a27fba4b4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Our usage of multitprocessing is problematic. In particular, there is a bug
in python 2.7 multiprocessing where signals are not handled until command
completion instead of immediately.
This adds a workaround into our wrapper function to deal with the issue.
(Bitbake rev: a16185e602b39b71475aa7e9ee80ad2b1f28d0f7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we trigger a shutdown whilst parsing for whatever reason, in some
cases we were not closing down the parser threads. This change
ensures we do so. The function names are not entirely intuitive
but the behaviour is more correct (and commented). The previous bug
with the stdout failure would trigger this one, if there was a cold
cache and parsing was required (but not otherwise).
(Bitbake rev: 25bfa2478f1c3a8eb695e1e5760e06db5be8f2fc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, IOErrors are just passed over due to the broken Exception
clause. A command like "bitbake X | <invalid command>" would break stdout
triggering a traceback. With these changes we print the exceptions, shut down
the server gracefully and exit which is a much nicer behaviour and is less
confusion to the user.
(Bitbake rev: 9544108f7b413038d871ce6ca88232de2f2434d9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if an exception occurs, we just run the idle handler again and
again, usually looping indefintely. Chances are the exception that occurred
will keep occuring and this is not a good place to be.
This was breaking the autobuilders with gigabytes of logs.
At least improve things so the cooker shuts down gracefully when this happens.
Some trace of the original problem may still be present on the console too!
(Bitbake rev: 1f28d8d3311262427938180435b68f0a35c2b330)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 000fa81013205dd9bc907ff7a61f06f57637212d)
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: fd293c5fcc89bc7a541d09e88a6aec5e0b2a1db2)
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 4e9aef14d747c37444a4fc683f9641906906afe9)
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its useful to have an event emitted when all of the sceneQueue tasks
have completed since the metadata can hook this for processing.
Therefore add such an event.
(Bitbake rev: 38d4f65bf1cbcdd5a2d60dff0e1d2859c34ed62e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The re module was imported at the top, no need to do it again.
(Bitbake rev: c29c2370635075949c75937c237602600f4f10dc)
Signed-off-by: Namhyung Kim <namhyung@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
so that it doesn't check it twice for non-empty events
(Bitbake rev: e304ace6d8f7d0808ee401b7c01146b2798a81b4)
Signed-off-by: Namhyung Kim <namhyung@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It turns out the codeparser cache is the bottleneck I've been observing when running
bitbake commands, particularly as it grows. There are some things we can do about
this:
* We were processing the cache with "intern()" at save time. Its actually much
more memory efficient to do this at creation time.
* Use hashable objects such as frozenset rather than set so that we can
compare objects
* De-duplicate the cache objects, link duplicates to the same object saving
memory and disk usage and improving speed
* Using custom setstate/getstate to avoid the overhead of object attribute names
in the cache file
To make this work, a global cache was needed for the list of set objects as
this was the only way I could find to get the data in at setstate object creation
time :(.
Parsing shows a modest improvement with these changes, cache load time is
significantly better, cache save time is reduced since there is now no need
to reprocess the data and cache is much smaller.
We can drop the compress_keys() code and internSet code from the shared cache
core since its no longer used and replaced by codeparser specific pieces.
(Bitbake rev: 4aaf56bfbad4aa626be8a2f7a5f70834c3311dd3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there is a corrupt/invalid cache file, we'd keep trying to reopen
it. This is pointless, simplify the code paths and delete the dead
file.
(Bitbake rev: c22441f7025be012ad2e62a51ccb993c3a0e16c9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we're writing out merged data to disk, its safe to assume that either
we loaded the data or couldn't. Loading it again is relatively pointless
and time consuming.
(Bitbake rev: 14c31e18e4a4a52ee54056aeefb09ff8c295b475)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch fixes a name collision on the event.data
in the MetadataEvent class. event.data is used in the
event system as a copy of "d" in certain situations,
and this collision triggered a bug leading to data loss.
[YOCTO #6332]
(Bitbake rev: 3f191b7cfe95aea4d4e96babf001d62d45dd3aaa)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 1cd369883469747a8158826bb8d67dcca2a8577f)
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The clearcase fetcher is used to retrieve files from a ClearCase
(http://en.wikipedia.org/wiki/Rational_ClearCase) repository.
Usage in the recipe:
SRC_URI = "ccrc://cc.example.org/ccrc;vob=/example_vob;module=/example_module"
SRCREV = "EXAMPLE_CLEARCASE_TAG"
PV = "${@d.getVar("SRCREV").replace("/", "+")}"
The fetcher uses the rcleartool or cleartool remote client, depending on which one is available.
Supported SRC_URI options are:
- vob
(required) The name of the clearcase VOB (with prepending "/")
- module
The module in the selected VOB (with prepending "/")
The module and vob parameters are combined to create
the following load rule in the view config spec:
load <vob><module>
- proto
http or https
Related variables:
CCASE_CUSTOM_CONFIG_SPEC
Write a config spec to this variable in your recipe to use it instead
of the default config spec generated by this fetcher.
Please note that the SRCREV loses its functionality if you specify
this variable. SRCREV is still used to label the archive after a fetch,
but it doesn't define what's fetched.
User credentials:
When using cleartool:
The login of cleartool is handled by the system. No special steps needed.
When using rcleartool:
In order to use rcleartool with authenticated users an `rcleartool login` is
necessary before using the fetcher.
(Bitbake rev: 76cff0aaad4cf10b325bceadd8ea90b3d75edcc2)
Signed-off-by: Dennis Meier <meier.dennis@siemens.com>
Reviewed-by: Roger Meier <r.meier@siemens.com>
Reviewed-by: Christian Liechti <christian.liechti@siemens.com>
Reviewed-by: Henrique Mendonca <henrique.mendonca@siemens.com>
Reviewed-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you hit Ctrl+C at the right point, the system processes the request
but merrily continues building. It turns out finish_runqueue() is called
but this doesn't stop the later generation and execution of the
runqueue.
This patch adjusts some of the conditionals to ensure the build really
does stop.
(Bitbake rev: 39b08c604ba713100e174c136f81f18eca6ef33d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patches fixes the build - project identification when
running under managed mode. The build is assigned to the
project from which it was triggered, and to the
build request, as to simplify relationships queries
in the database.
(Bitbake rev: af1d3373706d365f9138caec110fcb20a5966b7b)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Issue: TA53702
It was observed that a sequence in a script such as
bitbake --server-only ..
bitbake --observe-only ..
bitbake <some target>
could generate events from the server to the observer before
the observer was ready to read the events, and the early events
of builds were consistently dropped. Adding a "ready" note in the
log allows the script to scan for that message before proceeding
to building a target.
(Bitbake rev: 9949948f92bbe2717a05a380d3df63a332d39c9a)
Signed-off-by: Dave Lerner <dave.lerner@windriver.com>
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There may be a need to set the user for the transport rather than the subversion
command itself. Add a parameter to the url to allow this.
[YOCTO #6475]
(Bitbake rev: 56c294dc30b6c2575b1cf904e26b8b8bef7677c2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a '\n' to the last line of the file to fix:
No newline at end of file
(Bitbake rev: 54f1359ed2e9d47980cd221b7b43ef56543fe06d)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Right now shared work signature data is saved to the non-shared directory
which is confusing to everyone including bitbake. Whilst its messy,
extra the stampbase data instead, which ensures the sig data is written
to the correct location alongside its corresponding stamp file.
(Bitbake rev: 7ae1d4844d9d3a76f86ef32c5a794e51e334e588)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add the build control functionality to toaster.
* The bldcontrol application gains bbcontroller classes
that know how to manage a localhost build environment.
* The toaster UI now detects it is running under build
environment controller, and update the build controller
database and will shut down the bitbake server once
the build is complete.
* The toaster script can now run in standalone mode,
launching the build controller and the web interface instead
of just monitoring the build, as in the interactive mode.
* A fixture with the default build controller entry for
localhost is provided.
[YOCTO #5490]
[YOCTO #5491]
[YOCTO #5492]
[YOCTO #5493]
[YOCTO #5494]
[YOCTO #5537]
(Bitbake rev: 10988bd77c8c7cefad3b88744bc5d8a7e3c1f4cf)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add support to specify a connection token in the command line
and in the environment variable BBTOKEN.
When a client registers to a bitbake server, that client
will have exclusive access to the server. The client is identified
by a server-supplied token. If a client terminates, we cannot
reconnect to the server as the token is lost.
This patch adds the capability to specify the connection token
in the command line for xmlrpc clients. This allows us
to have bitbake work as an already-authenticated
client with the server and resume sending commands to a server.
(Bitbake rev: db5390940c0afbcdc9fbcf1225761968ae51d4a7)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
"set1 in set2" doesn't do what you'd expect, and if it did do a subset test
that's not the logic required by contains_any().
Instead get the intersection of checkvalues and val and check if the resulting
set is empty or not (by evaluating it in a boolean context), which tells us if
there are any elements in common.
Based on a patch by Valentin Popa <valentin.popa@intel.com>.
(Bitbake rev: 2e742c03e8dfdfa67899e7f5d579ed14bd87e139)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
contains_any is buggy so write a test case to demonstrate this.
(Bitbake rev: 25d10c349aab77bf99745c0a90dd8f9b90abccac)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This gives us an easy way to find out which recipes have been excluded
from world when there are many layers.
(Bitbake rev: cf33d4bfc53ae8dc6353fa295e1acd789d4fc2d2)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously, the buildinfohelper only set a django settings module
environment variable if none were set.
This may lead to problems when the starting the toasterui
from an already existing Django environment.
As such, we always override the variable to provide the
correct name for the local Django settings module.
(Bitbake rev: 8271e61a2fbddd3fc49556829675478d7505d58f)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We disable bitbake self start to prevent race condition.
ToasterUI will shutdown the server when the build is done
if running in managed mode.
We fix usage of kill server flag in the bitbake binary.
(Bitbake rev: 30159dbda3a40fa596302f91c705cb5f148c97a9)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The registration of a remote UI event handler may fail
if the server cooker is currently in some certain states.
This may happen, for example, when a remote UI is started
very fast after the bitbake server is started, and the
server hadn't time to finish initial configuration parsing.
Rather than fail outright, we have the remote UI event retry
registration for five time at one-second intervals,
in the hope it will succeed.
(Bitbake rev: c3d520c92ae4ae80d31926a416456df510654b6a)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we use the XMLRPC client API to connect to a bitbake server,
we want to receive errors from the API instead of having the
API exiting without warning.
Thus the "bb.fatal" calls have been replaced with "bb.warn" calls,
and we re-raise the original exception for handling by the
original caller.
The bitbake starting script has been modified to properly test
for failures in calling the client API and handle them.
Additional error handling added in the client, as to prevent
fatal crashes.
(Bitbake rev: eb63f08c33644f64752aaae2146a000956ce894a)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The configuration reading code should live in the
main bitbake entry point, and the server modules should
be supplied with correct configuration instead of attempting
to parse from configuration files.
This patch moves the endpoint address reading from XMLRPC
to the bitbake main script.
(Bitbake rev: ac5753274ff932e1d6f073ab4dab7bd6fe5355a1)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The big warning printed when people use -f is easily ignored/forgotten.
To raise user awareness, print a warning any time we include a tainted
stamp file into a build instead.
(Bitbake rev: 18f9bcbad059608e22fca20309314e1c399acec7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Checking for explicit exception names is bad, we also want to be able top
rely on inheritance. Fix these checks to be part of the real except clauses
so SkipPackage is recognised as being inherited from SkipRecipe.
(Bitbake rev: b131229145e1f2c372d6230a7b554e436c13c3f9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With new bitbake UIs having the cooker exit at 'random' points
in the codebase is problematic. This patch raises an exception
which matches the situation instead.
(Bitbake rev: 181a9735d02ebd517378558e909efc8b1b118973)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the depths of time we were rather confused about naming. bb files
are recipes, the event to skip parsing them should be SkipRecipe,
not SkipPackage. This changes bitbake to use the better name but
leaves the other around for now. We can therefore start removing
references to it from the metadata.
(Bitbake rev: 98d9e6e0f514a7cb7da1d99bf4bd5602b89426d6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need BBHandledException events to be passed through to the higher
layers, they don't need addition of any traceback since they've already
been reported to the user.
(Bitbake rev: f63b61f8997862439519f474cc09f3e05e95288c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we see errors during parsing, firstly its bad to show a traceback
for an expansion error so lets suppress this.
Secondly, raise a BBHandledException instead of a SystemExit to show
we've informed the user about the condition (printing a traceback in the
default unknown case).
(Bitbake rev: e01988d9a1b7c40e31161c6ce7b85c4405671068)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If an event handler triggers a SkipPackage event, we really want that
event to be received and processed by the higher code levels. Currently
it was getting caught and ignored which was leading to recipes
being present when they clearly shouldn't have been.
In general this exception catching looks to be doing the wrong thing. It
was introduced in
http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/bitbake/lib/bb/event.py?id=37cb4cc02b2e2b6c338c5943747e0a1ef15176b3
but we likely want exceptions to pass through to the higher layers.
This patch therefore removes that code.
(Bitbake rev: 79211b3434855942f9fad4a1db69ce7be911327c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If an ExpansionError occurs during better_exec() we should just raise it
instead of printing the traceback, so that recipe errors (such as broken
URLs in SRC_URI) are more easily comprehensible.
(Bitbake rev: 5b0da8932c318813138c113d2bb20498145dbd42)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Handle the following situations in a URL (e.g. in SRC_URI):
* Trailing semicolon in a URL - this is now ignored.
* Parameter specified with no value (no equals sign). This still
produces an error, but at least it is MalformedUrl with a proper
message rather than "ValueError: need more than 1 value to unpack".
(Bitbake rev: bfd13dfbc4c9f1dd8315002271791b1d9e274989)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A fix for the former patch when checking out a repository with
username and password using HG
(Bitbake rev: 0e7b594ccbceb3149f38776cea204807031ef69f)
Signed-off-by: Volker Vogelhuber <v.vogelhuber@digitalendoscopy.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you try:
Y = ""
Y_remove = "X"
in OE-Core, bitbake will crash with a KeyError during expansion. The reason
is that no expansion of the empty value is attempted but removal from is it
and hence no varparse data is present for it in the expand_cache.
If the value is empty, there is nothing to remove so the best fix is simply
not to check for None but check it has any value.
Also add a test for this error so it doesn't get reintroduced.
(Bitbake rev: af3ce0fc0280e6642fa35de400f75fdbabf329b1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ensure there is default command available if FETCHCMD_p4 isn't set.
Also clean up the getVar references whilst here.
(Bitbake rev: d150226d11d5f041f78c8c3ce4abc5465dbc81d8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The file inheritance logging has been here since the dawn of time. It
duplicates output many times over and logs to debug level 2. When running
with the debug option, its understandable the user may want to see the
paths of files included in the build.
These changes remove pointless/duplicate output and print the
include paths clearly at debug level one in a form which
users should be able to more easily parse.
(Bitbake rev: 633a56987df639295fe9128418da5634050063e5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a user runs with the -v or -D options, its understandable they'd
expect to see log output from the workers yet right now a bug in the
log handling does not show this.
Fix the conditional to ensure such log output is shown on the terminal
when it has been requested. Ideally this data should always flow to
the logfiles but that is for another patch.
This also fixes the code to do what was always intended in the comments,
i.e. if the user specifies -D or -v, output is shown from the tasks,
otherwise notes are suppressed.
(Bitbake rev: 20a3c93d8572969e76563f29bff89400b93ffae7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It was reported that bitbake -D made no mention of which append files it
was using. bitbake -DD does but it makes sense to increase the log level
of this piece of debug information.
[YOCTO #6262]
(Bitbake rev: 5824bf9c6feea05567d155911f4ab2e371911d34)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 42b7be1a6c5ff559f4a213eed04b370ca955460d)
Signed-off-by: Nick Lewis <nick.lewis@usa.g4s.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Unfortunately we've been neglecting to pay the correct tributes to the
cookie monster and hence the datastore is malfunctioning.
Currently tributes are only paid on the last part of a variable after
the last "_" character. We need to split by *all* "_" characters since
an override may contain the character.
This fixes the code so the correct number of tributes are made. Paradoxically
parsing appears to be faster after this change.
(Bitbake rev: d1c712fd3a59fa804e6fd451612c30487671f3a2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the power were to fail, it doesn't matter to us much if the data
makes it to disk or not, we'd have other problems. However an fsync()
call on a multi build autobuilder is painful so lets avoid them.
This is particularly true in this case if a timeout causes a reconnect
during a build.
(Bitbake rev: ec28256ac2a30f047585e8f61200d764bc295ded)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if you do localdata.getVar, the code parser simply ignores
the references. Change the code to use endswith() to catch more of the
references. These names are probably unique enough to get away with this.
Bump the cache version to ensure things get updated.
(Bitbake rev: cf763cddc3faa2361b4c4dbd08419e4ebabf208f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This includes contains_any in the special handling code for sstate. It
does not take into account the equivalence of the values. In current
code, considering 'bb.utils.contains_any("A", "foo bar", ...)':
A = "foo"
A = "bar"
A = "foo bar"
All those will get different signatures.
(Bitbake rev: d1e3345d715e488ec3f5515fb0e1fb39366346bc)
Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Doing so was causing leakage between the execs of the main value and that of
the subshell value, and was causing the cached subshell value to be used for
the overall variable. At the least this could cause execs contamination
between two variables that while differing, run the same subshell. Beyond
that, it's possible we could have been using an incomplete cached value of
a subshell for that of the main value.
Before this, bb_codeparser.dat would change between parses with differing
bbfile parse order. After, it does not change.
The codeparser cache version is bumped, to ensure we don't use potentially
incorrect cached values from previous runs.
This should hopefully resolve the difficult-to-reproduce issues we've seen at
Mentor Graphics where bitbake emits a script to run a task and misses
dependent functions, resulting in 'command not found' failures. This issue has
also been mentioned on the oe devel list, where someone hit a case where
oe_runmake was missing from a do_install task (IIRC). Adding debug information
showed that bitbake's information about the variable dependencies for this
task is inaccurate in the failure cases.
(Bitbake rev: 97537e4786a1e3a329249497498b59b8f5174fc3)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The initial version of this patch was reverted in
0a94e568152de550dedc8135a766beb18bf064ab
However, it was later agreed upon that it was reverted due to
a misunderstanding during code review; the patch is ok.
This version of the patch also removes an unneeded initialization of
the 'checksum' variable outside the scopes it is being used in.
(Bitbake rev: 9ee19a3ca2f8e11a91f0289ea3486310c61d40f2)
Signed-off-by: Jacob Kroon <jacob.kroon@mikrodidakt.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Following up a previous patch for mercurial fetcher, I just fixed a
problem when calling update on a repository with subrepositories enabled.
Attached a patch that fixes this problem
>From caae519a2bd64bf7c729bb26aff344827def47fb Mon Sep 17 00:00:00 2001
From: Volker Vogelhuber <v.vogelhuber@digitalendoscopy.de>
Date: Tue, 29 Apr 2014 15:29:47 +0200
Subject: [PATCH] bitbake: - fixed authentication issues in case of using sub
repositories
(Bitbake rev: dfa041c940caad12da50126a559afc8de089eeda)
Signed-off-by: Volker Vogelhuber <v.vogelhuber@digitalendoscopy.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit 6dfb33fd7204e09815bde7ba5bf21941a73fe965.
It causes an unindeded change in behaviour.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Cleanup the fix done in f9416e76e272ba3249abff099f6f3a47fe82e03e.
Instead of adding continue statements we can just move the last
statements into the final else-clause.
(Bitbake rev: 6dfb33fd7204e09815bde7ba5bf21941a73fe965)
Signed-off-by: Jacob Kroon <jacob.kroon@mikrodidakt.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We only ever clone other repositories, if there were a problem such as power
failure, we'd blow away data and rebuild. As such we don't need fsync(). With
filesystems like ext*, the fsync pushes nearly all the data out to disk
which impacts all running processes.
We therefore set a configuration parameter to disable the fsync() calls.
Also fixup a case where basecmd wasn't being used for no good reason.
(Bitbake rev: 0a26abaf3a1e34d556c9375068dd17c879568d0f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The optimisation where only the data we're interested in was finalised
was good but it turns out we can do better. In the case where a
class-extension is to be targeted, we can skip the other targets.
This change does that and speeds up parsing at the bitbake-worker
execution time. Specifically, you can see an improvement in the speed
of bitbake X -n.
(Bitbake rev: b56918c7ef7913e84356c69ee9b269844a446728)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In dry run mode, stamps for noexec tasks are being written out which
is incorrect. Avoid this.
(Bitbake rev: aa6448a0552ba2947ac262b8b5314a593d1058d3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When looking at a list of tasks, do_patch and do_unpack were being
given equal priority when one clearly depends on another. The
reason for this was the default task weights of 0 being to tasks.
This is therefore changed to 1 to allow correct weighting of dependencies
which means the scheduler has better information available to it about
tasks.
Weight endpoints differently (10) for clearer debugging of priorities.
(Bitbake rev: 12dc1d17fac3e8ec420fcafb06186d32fd847d89)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The zero priority task should be run first but was being confused with
the None value the priority field defaulted to. Check for None
explicitly to avoid this error.
In the real world this doesn't change much but it confused the debug
output from the schedulers.
(Bitbake rev: 49c9d8c9400f74c804c2f36462639236e0841ff0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since we now get wildcards in the file checksum list in the cache, we
need to ignore them when checking to see if they still exist. This
fixes connman-gnome reparsing on every bitbake execution in OE-Core.
(Bitbake rev: 340c250fc664414ab2715a454bedbd19e8efe103)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add some explicit tests for unpacking local files to the appropriate
location. Some of these tests are actually testing for broken behaviour;
these have been called out in the comments, and associated bugs have
been filed.
(Bitbake rev: ca921c773c52392a5a338b2f493ad38c8132f708)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>