Add some explicit tests for unpacking local files to the appropriate
location. Some of these tests are actually testing for broken behaviour;
these have been called out in the comments, and associated bugs have
been filed.
(Bitbake rev: ca921c773c52392a5a338b2f493ad38c8132f708)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the recent change to split the log output to stdout and stderr,
error messages that appeared while the footer was printed got all
messed up. This was because the messages to stderr was output _after_
the footer, then clearFooter() tried to remove the footer but removed
the error message and parts of the footer.
(Bitbake rev: 4fafea4fa69542b491e84463f6eae0d5bf645673)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there is a directory matching a wildcard in SRC_URI when getting file
checksums, we should recurse into that instead of producing an error.
(Bitbake rev: ae87b7eb414e3d5eefd2effec7b30c22d2186b02)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The Local fetcher's localpath is returning the parent directory for a
wildcard match; we need to handle this and add the wildcard
specification so that we checksum the correct files.
Fixes [YOCTO #6127].
(Bitbake rev: 4a90edd62c16cbf41b5b93280e155077564c774a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The path for "recipes/images/" was not added in BBFILES when Hob had
to search for an image recipe. Therefore, it could not find it and an
error occurred.
This path needs to be added when Hob is launched.
[HOB #6086]
(Bitbake rev: 35c67281775b08925957c32663d587d486944e0e)
Signed-off-by: Irina Patru <irina.patru@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously, if a file listed in SRC_URI was missing at the time of
parsing, and then was added and bitbake run again, it would not be
picked up because the recipe was cached without it. If we allow the file
to be added to the list of files to checksum, then it will be checked
for and found on the second run.
Fixes [YOCTO #4790].
(Bitbake rev: 71da822762cb298261cccdfa54b9c0fea02c3c5d)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For a while its been puzzling me why connman-gnome rebuilds as often as it
does. It turns out you can trigger this with a new checkout of the metadata.
The SRC_URI that is causing the problems is:
SRC_URI = "file://images/*"
and rather oddly the results in checksums for a file "." being added to
the tree, e.g.:
('.', 'ab48a68186f0e0f277c21ef4cb390b4b')
The problem is that when iterating files lists, the checksum variable can
become set yet we don't break the out from the for loop, which leads to
odd (and non-deterministic) entries being added into the file checksum list.
The exact item added probably depends on the order of items on the disk.
Before this change, bitbake-diffsigs on connman-gnome:do_fetch would report:
This task depends on the checksums of files: [
('connman-signal-03.png', 'f6c16aee57b37b73793a2f1dea433ffa'),
('connman-signal-02.png', 'ad0cd22710c097d8174121fc1023c3be'),
('connman-signal-01.png', '8842bd83d2fa9ba56480df34c727c629'),
('null_check_for_ipv4_config.patch', 'a23271e41c9fe81551244d875106af10'),
('connman-signal-05.png', '808589e7e8d502b44c7b007e9e68d48c'),
('connman-signal-04.png', 'ab48a68186f0e0f277c21ef4cb390b4b'),
('null_check_for_ipv4_config.patch', 'a23271e41c9fe81551244d875106af10'),
('0001-Removed-icon-from-connman-gnome-about-applet.patch', 'e2d8269357c1e8c84291606da24eea85'),
('0001-Removed-icon-from-connman-gnome-about-applet.patch', 'e2d8269357c1e8c84291606da24eea85'),
('.', 'ab48a68186f0e0f277c21ef4cb390b4b')]
Afterwards:
This task depends on the checksums of files: [
('connman-signal-03.png', 'f6c16aee57b37b73793a2f1dea433ffa'),
('connman-signal-02.png', 'ad0cd22710c097d8174121fc1023c3be'),
('connman-signal-01.png', '8842bd83d2fa9ba56480df34c727c629'),
('null_check_for_ipv4_config.patch', 'a23271e41c9fe81551244d875106af10'),
('connman-signal-05.png', '808589e7e8d502b44c7b007e9e68d48c'),
('connman-signal-04.png', 'ab48a68186f0e0f277c21ef4cb390b4b'),
('null_check_for_ipv4_config.patch', 'a23271e41c9fe81551244d875106af10'),
('0001-Removed-icon-from-connman-gnome-about-applet.patch', 'e2d8269357c1e8c84291606da24eea85'),
('0001-Removed-icon-from-connman-gnome-about-applet.patch', 'e2d8269357c1e8c84291606da24eea85')]
which is correct and deterministic without the "." entry.
(Bitbake rev: f9416e76e272ba3249abff099f6f3a47fe82e03e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some warnings and errors may come from a non-build context,
and they were not saved, even if they were counted for the current
build.
This patch saves these messages in memory until we have the
entire build context available.
[YOCTO #5642]
(Bitbake rev: fc7a74e7961775b5d7ff25298abed10138d24dc9)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We rename a local variable in as to prevent a conflict with
a similary named function parameter.
(Bitbake rev: f4e57f794651c4894600445e843ca9d5e104cd84)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This solves a problem of custom images which inherit a base
image with IMAGE_BASENAME overwritten in their recipe by a
different value than its default one: ${PN}.
The value of IMAGE_BASE causes a crash when hob will try to
create symbolic links to the resulting images from the deploy
directory, because it will look for names similar to
<original_recipe_name>-edited-timestamp-machine.rootfs.*
which might be different from the actual resulting image.
The solution is to simply overwrite IMAGE_BASENAME in the
custom recipe to the default value in the case IMAGE_BASENAME
is found in the base recipe.
Some recipes which were affected by this problem are those
from meta-fsl-demos (e.g.: fsl-image-test).
[YOCTO #6017]
(Bitbake rev: e42ee93519000f827be49659b6b5fb7717b3d592)
Signed-off-by: Marius Avram <marius.avram@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fixes a problem which happened when you tried to build an
image by selecting 'Start with an empty image recipe' from the
Image configuration page of hob.
The reason on of the bug was that the name of the resulting image
was threated the same way as ordinary custom images, when in fact
they should use the default name: "hob-image", because they do not
derive from any other recipe.
[YOCTO #6102]
(Bitbake rev: fa4ea3b4b40e7e9e6767e0cd51c6701e0af07135)
Signed-off-by: Marius Avram <marius.avram@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To find if a recipe is for an image, Hob checks if it inherits
image.bbclass. But when you add testimage in local.conf, this will
be added for each recipe, and it pass the test. Adding a "/" before
"image.bbclass", will check only for image.bbclass.
[YOCTO #6117]
(Bitbake rev: be8511c9d474c570f6ca7078e28919c8a5175a42)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you are using the testimage class then the old test mistook the
global inheritance of testimage.bbclass with the recipe inheriting
image.bbclass because it was only looking for that at the end of the
string.
Also tidy up the code so you an easily tell what it's doing. (The
original method may have been more "pythonic", but it does nothing for
readability.)
(Bitbake rev: b05e741cb5fe44b37538f2b727782f80dc9bb8fa)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These were not being collected properly because we were explicitly
excluding variables defined as functions from being stored in the
database. We don't want these to be shown in the variables list, and in
any case it makes sense for these to be stored elsewhere, so create a
separate model to store these.
Fixes [YOCTO #6050].
(Bitbake rev: 0d76a5461ce4bd554ff70a465064969e53edf0a4)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch makes sure that the setFeatures command is marked
as read-only and that it can only run if the cooker is in
the initial state.
Additionally, remove logging from the XMLRPC module in favor
of sending the exception to the client for easy processing.
[YOCTO #6089]
(Bitbake rev: f0a1a3e24757f7658d272035620465f92a3e4c3c)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is an update on the sstate file saving data.
It saves both found and missed sstate tasks.
(Bitbake rev: 60c577b1080219b795d3c8ab4e149e929cf9ce14)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The first part of the sstate code checks en-mass whether given checksums
are available. The next part of the code then either triggers those
setscene tasks either running them or skipping them if they've been
covered by others.
The problems was that this second part would always skip a task if it
was unavailable in the first part, even if it would have otherwise been
covered by other tasks.
This mean the mere presence of an artefact (or lack of presence) could
cause a different build failure.
The issue reproduces if you run a build and populate an sstate feed, then
run a second build off that feed, then run a third build off the sstate
feed of the second build (which is reduced in size).
The fix is rather than immediately skipping tasks if the checksum is
unavailable, create a list of missing tasks, then, if that task cannot
be covered by others we can skip it later. The deferral makes the
behaviour the same even when the cache is "incomplete".
[YOCTO #6081]
(Bitbake rev: 5edb1a3e3f454ba6e65551174d86229db2f99636)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a setscene task has [depends], its possible they may still get executed out
of order. The issue is that the dependencies are set to set() for all tasks
involved. This patch adds back in explict dependencies within these chains
to avoid the setscene task failures.
[YOCTO #6069]
(Bitbake rev: 724c889eed3b03d3199810c185086d3973af826c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than passing featureset around various places where the data doesn't
really belong, run a command at connection time to set the appropriate
features. This is similar to what the process server does.
(Bitbake rev: c3b5cc5691291c74dd315c4439c80e0e4b2b5c1d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the featureset didn't match the defaults, we'd pay the price of two
base configuration parses which showed up adversely in the performance
benchmarks. This also passes the feature set into the cooker creation so
in the common case we don't have to reset the server. This speeds up
both knotty and hob startup. If the featureset doesn't match, the system
will reset as before, this just streamlines the common case.
(Bitbake rev: 1249543c4dbf3edeac033d888497864cfc807a4e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the change to more optimal default featureset behaviour, a race was
exposed by hob where the code may try and change self.data before it
exists. This change avoids that.
When the datastore is created, the cooker configuration is used so
data tracking is correctly handled regardless.
(Bitbake rev: 9d8f7efbc39d64124936ccaeb3c47a112e595d78)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its not immediately obvious to the user that a logfile exists for a failed setscene
task. Add code to knotty to display where that logfile is in those cases.
[YOCTO #6055]
(Bitbake rev: 0664fa15597785dd90cf205531a9801e6da6ba47)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using bitbake -e in scripts, it would be helpful if the error
output appeared on stderr, not stdout. This change enables that building
upon the new bb.msg filters.
[YOCTO #5274]
(Bitbake rev: ebb797fc5c37d729e3cc8b2dc7156287d385c13b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I can understand why some programs would want the progress on stderr so
that real output can be captured on stdout. This is confusing for bitbake
since we don't show a progress bar at all in non-interactive cases.
Therefore make sure the progress bar goes to stdout, not the stderr default.
(Bitbake rev: 0529aa9966df5c56b07affe865efce18852efe5a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add logging filters which can be used to split output between stdout and
stderr, ERROR messages and above as passed by the Err filter, anything
below ERROR is passed by the Out filter. This is useful when trying to make
stderr more useful.
(Bitbake rev: d3e1419ef76be5e9ec976597361a5e14a7b6bcb6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sanity tests are currently a pain as its hard to control when they run. This results
in issues where for example the bitbake -e output is not useful as the sanity tests
prevent it from executing. The sanity tests should run later than the base configuration.
This patch changes the sanity tests to always be event triggered with the option of
returning either events on the status, or raising errors. A new cooker feature is used
to change the behaviour depending on the controlling UI.
This does need a change to sanity.bbclass in the OE metadata but its worth the pain
for the increased flexibility and control this offers UIs and the improvement to the
user experience.
(Bitbake rev: 32e171bcc92c6e27fefee971e8714ddf8e1a8ac1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows the commandline options to be processed in the dump signature
code.
(Bitbake rev: ef8537a2e9b48f4fe065a165c102935aee2c9029)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is no easy way to make this change. We really need parameters for the -S
(dump signatures) handling code. Such a parameter can then be used within the
codebase to handle the signatures in different ways.
For now, "none" is the recommended default and "printdiff" will execute the
new (and more expensive) comparison algorithms.
(Bitbake rev: b9873588696507dfb6aade6821f6f75cb9a19e0a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
FOO = "foo bar"
FOO_remove = "bar"
FOO_FOO = "${FOO} ${FOO}"
would show FOO_FOO = "foo foo bar" rather than the expected "foo foo".
This is actually a cache bug, this patch ensures the right value is
put into the cache. The preceeding patch adds a test case to ensure
we don't regress in future.
[YOCTO #6037]
(Bitbake rev: 2a80735183e8faa110b4c6d8d85c4707f28e03a1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We currently don't have test cases for _append, _prepend and _remove. This
patch adds some basic tests and includes a test case for a recently reported
issue with the _remove operator.
(Bitbake rev: 93291bd90e18808c7a1c8c692949396bbc7e4348)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The runqueue should be using the "realtask" ID to lookup the task
hash, not the "task" ID. This patch resolves corruption issues where
incorrect task hashes were displayed within toaster.
(Bitbake rev: 84be1a27f89d1bf63c21f06d831df0a66a5db860)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The function is used to save additional variables in the configuration file
when the user adds a new (key, value) pair from the Settings->Others. There
was a problem though when the function was trying to retrieve an older
instance of EXTRA_SETTINGS from the configuration file. Sometimes its value
was returned as a dictionary and sometimes a string, which caused a crash when
calling ast.literal_eval(). The reason of the problem must be a change in
bitbake's parsing system. The changes will fix this issues.
While analysing this problem I discovered that the variables were not saved
properly in the configuration file after consecutive changes to Settings->Others
because of the way saveConfigurationVar() from cooker.py works. This patch
will also solve this issue.
[YOCTO #5989]
(Bitbake rev: bdbcd8866104c315fc9da631407d4280433dbfde)
Signed-off-by: Marius Avram <marius.avram@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When a variable was removed from a configuration file it was not
removed from memory. This also had the effect of not allowing
to set a new value for the same variable with saveConfigurationVar.
(Bitbake rev: 30cd1fab6633aaf50ef53eefccc6d69d598eb293)
Signed-off-by: Marius Avram <marius.avram@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds the capability to have the Toaster UI
detect when the Bitbake server exited and cleanly
trigger a clean shutdown of the system through the toaster
starting script.
(Bitbake rev: a9cfa3eacfc99550e1ad3f8bb61b2a0bc9b44332)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We save the missed sstate tasks as tasks that executed
but have the sstate_result set to "SSTATE_MISSED", signaling
that the attempt to find an sstate file failed.
(Bitbake rev: 6f22e02614adcc642fe011e5e31ca4936d1cb19d)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When saving task stats, if there were multiple tasks executed
based on the same recipe file, we might have saved the stats
to the wrong task by selecting another recipe.
This patch takes the PN into account to properly select
the file stats.
A check is also made to make sure we don't fail saving
data due to interrupted builds.
(Bitbake rev: e855031410daf2b99a6ca40b70956fe67c96f71c)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
File_name information of Target_image_file is being collected for a
.rootfs.manifest file. We would like not to collect this. The solution is
to cross check the information gathered for file_name with the content
of the IMAGE_FSTYPES variable. If any of the file_name entries does not
match the content of IMAGE_FSTYPES, we do not store it.
[YOCTO #5189]
(Bitbake rev: 017771ed0508b247edaf875789260906f44381f4)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a recipe for an image was edited from the hob interface the name
of the files outputed in the <build_dir>/tmp/deploy/images/${MACHINE}/ and the
temporary recipes from <build_dir>/recipes/images/ contained only the
generic name "hob-image". From now on both the temporary recipes and
the output from the deploy/ directory will contain the name of the base
recipe appended by the "-edited" suffix, in the case when a base image recipe was
edited. The base recipe can be a standard recipe (e.g core-image-minimal) or
a custom created and saved by the user.
For example, if core-image-minimal is edited the deploy/ directory will contain
core-image-minimal-edited-20140318-140428-qemux86.ext3 and the recipes/images/
directory will contain the recipe core-image-minimal-edited-20140318-140428.bb.
[YOCTO #5002]
(Bitbake rev: f34575809677dc52e1071a3ae3daebe92819cec0)
Signed-off-by: Marius Avram <marius.avram@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When setting the username after already having set the password, the password
was unexpectedly reset. This change fixes this issue and introduces unit tests
to make sure it doesn't happen again.
(Bitbake rev: 25faef3a047f9c7564089463d7c96f6910b640cb)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a case in meta-intel where a SRC_URI contains both a query string and
URI parameter:
https://edc.intel.com/Download.aspx?id=6190;downloadfilename=LIN_IEMGD_1_14_GOLD_2443.tgz
Python's urlparse thought the URI parameters were part of the query parameter
value, but in the bitbake context this is obviously not the case. As bitbake's
usage isn't really RFC compliant, we have to extract and remove the URI parameters
*before* urlparse sees the URI.
(Bitbake rev: c2f27ae4271985b48f957747b6ea372c8699cb49)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This change introduces the .query property of the URI class. It is a
read/write dict of the parameters supplied in the query string of the
URI. E.g.:
http://example.com/?foo=bar => .query = {'foo': 'bar'}
(Bitbake rev: 1cb2b3c458c8c5521591d2c8f2e0058143fc77bb)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch restricts the elapsed calculation to just
the events that have the "time" parameter set.
This fixes an error where data was lost due to an exception
where invalid dictionary lookups were made on the wrong
events.
(Bitbake rev: fa9f4eb8784553deb782bff34c5e04012c2c52c9)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The buildstats and toaster use separate time markers to measure the
duration of task execution. This causes a mismatch in the time
measured by buildstats class and the time measured in toaster.
The solution implemented here is to timestamp the creation of
every TaskBase event on the bitbake server side and calculate
the execution duration as the difference between creation time
of TaskSucceeded and TaskStarted events.
Based on an original patch by Marius Avram.
[YOCTO #5485]
(Bitbake rev: 7a08282c074c264f414cf7665dc873f51245072c)
Signed-off-by: Marius Avram <marius.avram@intel.com>
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When an evaluation was made for a configuration file the path to the
file was saved as a relative one. The change in this commit will save the
location as an absolute path. This way the user will have full information
regarding the location of the file where a variable was changed and the
line withing the file.
[YOCTO #5562]
(Bitbake rev: df9e22901555b06fef308f7136547f2c47ccec35)
Signed-off-by: Marius Avram <marius.avram@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use of waitpid on the worker processes is a bad idea since it conflicts
directly with subprocess internals. Instead use the poll() method
and returncode to determine if the process has exitted, if it has,
we can shut down the system.
This should resolve the hangs once and for all, famous last words.
(Bitbake rev: 60969cd62e21e7d4af161bf8504b7643a879c73f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're running into processes using 100% cpu. It appears theses are locked in
a subprocess.poll() type loop where the process has exited but the code is
looping as its not handling the ECHILD error.
http://bugs.python.org/issue14396http://bugs.python.org/issue15756
This is likely due to one or both of the above bugs. The question is what actually
grabbed the child exit code as it wasn't this code. Its likely there is therefore
some other code racing and taking that code, it may be some kind of race like:
http://hg.python.org/cpython/rev/767420808a62/
where the fix effectively catches the childs codes in a different part of the system.
We could try and get everyone onto python 2.7.4 where the above bugs are fixed however
for now its safer to admit defeat and go back to polling explictly for our worker exit
codes.
(Bitbake rev: 5b9a099ec2a1dc954b614e12a306595f55b6a99e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Catching all child exit status values is a bad idea. Setting an http sstate mirror
is a great way to view that spectacularly break things. The previous change did
have good code changes so don't revert those parts.
(Bitbake rev: fa7ffb62d510ac1124ae7e08fa4d190a710f5b54)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are several problems. Firstly, a return value of "None" can mean
there is a C signal handler installed so we need to better handle that
case. signal.SIG_DFL is 0 which equates to false so we also need to
handle that by testing explicitly for None.
Finally, the signal handler *must* call waitpid on all child processes
else it will just get called repeatedly, leading to the hanging behaviour
we've been seeing. The solution is to only error for the worker children,
we warn about any other stray children which we'll have to figure out the
sources of in due course.
Hopefully this patch gets things working again properly though.
(Bitbake rev: 973876c706f08735c1b68c791a5a137e5f083dd2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Failures on the autobuilder look like this handler is recursing. That
shouldn't be possible but it doesn't hurt to code as such.
(Bitbake rev: e39e85803cbe1ef9413a118868c19087c0546d01)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've noticed hanging processes which appear to be looping around
waitpid. Its possible multiple calls to teardown are causing problem
or in theory multiple registrations (although the code should not
allow that). Regardless, put better guards around signal handler
registration.
(Bitbake rev: 79acfb0853aa3215215cee89a945f8e97b0a8fae)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Showing "Deploy image successful" after 'dd' returns
may determine the user to disconnect the usb stick even
though the writing operations are not finished.
This patch makes sure that the entire image is deployed
on the usb stick before the user is informed about any result.
[YOCTO #5892]
(Bitbake rev: cc98b19112ab875ebc7cb604cd96acadac4cbf21)
Signed-off-by: Valentin Popa <valentin.popa@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we didn't setup any workers (such as bitbake -S), this would error
since we're trying to set a signal handler to None. This patch
avoids that problem.
(Bitbake rev: ce17478c8197abf178c00774f5bbe23fd4375ee2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch improves the recipe matching algorithm for
for matching recipes for native tasks.
(Bitbake rev: c350e4924abab8688c539608fd7f3af687d7265a)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patches fixes the inode type saved when writing the
target file list information.
(Bitbake rev: 9f34a1c5e94d73cdba1def7059c60211514e054c)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes an issue in hob which happened when the local.conf file was
modified externally by appending "eclipse-debug" to the IMAGE_FEATURES
variable. The reason of the problem is that some IMAGE_FEATURES are
not available in the image.bbclass file and they are declared in the
core-image.bbclass. Now a default hob image will inherit core-image.
[YOCTO #5711]
(Bitbake rev: 81413d94f40f58d790d7a7dc4259108f9c5d4fc0)
Signed-off-by: Marius Avram <marius.avram@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Based on image_name, the target is obtained, and the path
is added to the database.
[YOCTO #5649]
(Bitbake rev: 911b5191133956c30d53f57629c115db196b9ac8)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using ImageFileSize Metadata event, the image output file and its
size are populated into target_image_file table.
[YOCTO #5189]
[YOCTO #5228]
(Bitbake rev: a0b06d362b9aa08fda293489467af343c6ca6de4)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The sigchld handler was reaping any processes and this was leading to
confusion with any other process handling code that could be active.
This patch:
a) Ensures we only read any process results for the worker processes
we want to monitor
b) Ensures we pass the event to any other sigchld handler if
it isn't an event we're interested in so the functions are properly
chained.
Together this should resolve some of the reports of unknown processes
people have been reporting.
(Bitbake rev: fe8baaa2f533db7a1b7203476c675588923d8d45)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding code to write files-in-image data from the metadata
event to the database.
(Bitbake rev: f3a69ef7cc536a4b879d60936199a1002584c4f4)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of a significant number of calls to waitpid, register a SIGCHLD
handler instead.
(Bitbake rev: 76029d08ad56a0a264ff9738a0336971a455b7f5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When idle functions finish, its likely we have some other work
to do, so don't sleep in the select call but instead, skip it.
This removes small amounts of latency in common commands.
(Bitbake rev: 069d6538f83b607cb46c6fe21bf6c596e8b99242)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the server exits, we no longer appear to need this delay. This
is likely due to improvements in the various exit codepaths. There
is therefore no longer any point in taking the latency hit.
(Bitbake rev: 8e75ee29ae07e13f23525c5c6045fbf6cdbe7675)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its not possible to notice the change of status of an Event() in
the select call we sleep in. It would be possible in python 3.3 but
for now use a pipe instead. This removes small latency when bitbake
commands finish since the system doesn't sit in the select call.
(Debugging these kind of issues is apparent by setting a long sleep
for the select call)
(Bitbake rev: def28239b0f0d5f1cf13214b263114a5328538b7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is no point in waiting 0.25s for when we should be processing
the shutdown. This simply reordering removes latency from the
bitbake command.
(Bitbake rev: f147b41bcaf9d05b5ba3a70100f1ca799979aee7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The continue statement was missing for this event and the event was then
listed in the "known safe to ignore list". Clean this up.
(Bitbake rev: c4ee342300bf905e6e3bef581c61b86289461536)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A run of "bitbake bash -c unpack" when the task has already been
completed resulted in about 9000 calls to logger.debug(). With this
patch which comments out some noisy/less usefull logging and moves
other logging calls outside loops, this number is reduced to 1000
calls. This results in cleaner logs and gives a small but
measurable 0.15s speedup. The log size dropped from 900kb to 160kb.
(Bitbake rev: d2677f084fe1d8846db77d89ef5e6ffb18dc171a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the cooker receives a SIGTERM it currently hangs using 100% CPU,
This patch adds in an intercept for the event and puts the cooker into
shutdown mode allowing it to exit cleanly/safely and avoiding the hang.
(Bitbake rev: 00c22434123739b0819b31d7b1d353901a3e12da)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently a SIGTERM to the UI process causes the UI simply to lock up.
By setting an exit flag, the waitEvent can raise a SIGINT, allowing the
UI to break out the event loop and exit. Currently this is results in a
traceback but that is more desirable than a hanging process.
(Bitbake rev: 0d12041eceeae6bba2034b04913bb13abd67bd15)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The use of a manager in the process server causes some issues since it remains
around for the lifetime of the server even though its only used during
initialisation and the system doesn't respond well to SIGTERM events
to the extra process (and two threads) the implementation involves.
Switching to a dedicated command simplifies the server process structure.
(Bitbake rev: 74532a7cf8ccea8b85f1cda5d5bc23d2f3c72a08)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a command to allow backends to set particular 'features' on the
cooker (server).
(Bitbake rev: f547d6ec6cfd677d71fa96dd3c69823c00dc6c69)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the worker has already gone missing (e.g. SIGTERM), we should
gracefully handle the write failures at exit time rather than throwing
ugly tracebacks.
(Bitbake rev: 1b1672e1ceff17fc4ff8eb7aa46f59fce3593873)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the worker (or fakeworker) process disappears for some reason, the
system doesn't currently even notice. To fix this, we call waitpid
periodically, looking for exit events of our children. If these
occur, we can gracefully shutdown the server.
(Bitbake rev: ee28ddadaa7ef91e4d4b7d22fc267382aaad6d01)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When an exception occurred, the terminal parameters (such as echo)
may not be reset correctly. This change ensures they do get
atexit time in all cases, avoiding the terminal corruption issues
that could sometimes occur.
(Bitbake rev: e1d89166f2dfe46412ff9a5610dd57b0cef74fe3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds extra checks when selecting and writing
task and recipe objects to the database.
The patch fixes several issues where tasks may have been
misidentified between virtual-native and target tasks,
or spurious task objects may have been created.
(Bitbake rev: a6e597e690b3c6c6fa2af6db8cd871c02fc80421)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch fixes a logical error when updating task information
based on the corresponding sstate task state.
(Bitbake rev: 777458a20a7f686881e525a4d81b286c486ead6a)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch updates the database schema to resolve a
number of issues discovered while implementing the
UI interface.
We do not expect that all the data will come in valid
at this point.
[YOCTO #5453]
[YOCTO #5833]
[YOCTO #5836]
[YOCTO #5811]
[YOCTO #5812]
[YOCTO #5820]
(Bitbake rev: f8ad96d10a095e21fd2ce424c45e17f54642fb54)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add assert statements that validate expectations of
correct values and context when the data collection code is running.
These checks will help pinpointing unexpected data or
call flows, reducing debugging time.
Also contains a couple of very small fixes discovered through
these checks, including a virtual:native conflict with
regular tasks.
[YOCTO #5553]
(Bitbake rev: e2fbd5c6fa6b53514e2cb23d42aa639020d8a475)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We update the structure used to hold interm task data,
before it is written to the database, to lower the changes
of key collision.
This will also lead to a cleaner data structure and easier
inspection.
(Bitbake rev: 49cb9f543526a161bc4c097f94422ea08b491ef9)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This optimization is in support of the bug #5485. The function called
at the beginning of every build: store_dependency_information was taking
approximately 20sec and it was delaying the arrival of events from the
event queue. The change minimizes the calls to _save_a_task(),
reducing the time to half.
(Bitbake rev: b86fd2be40303d886fdb9ad3009355584d285acc)
Signed-off-by: Marius Avram <marius.avram@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to mark the package entries that are
created solely for dependency tracking purposes.
In order to avoid altering the database schema,
we mark the dependency targets with size = -1, since
this is not a valid size anyway and makes for easy
filtering.
[YOCTO #5803]
(Bitbake rev: d11ed273dd6c520b16e9ccfe79476f340006a55d)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to separate tasks with invalid states from the
no exec tasks, we add a new value OUTCOME_EMPTY for the tasks.
OUTCOME_EMPTY has the same value as OUTCOME_NA as to maintain
compatibility with already existing builds. New value for
OUTCOME_NA can be used to detect tasks with invalid states, i.e.
it should never appear after finishing a build.
Fixing noexec tasks outcomes.
[YOCTO #5763]
(Bitbake rev: 475643ad78796835bf2e731b9d0fa5794ec80dd1)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some of the previous cleanups broke the downloadfilename functionality.
This change fixes the code to ensure the commandline is correctly built.
Thanks Kristof Robot <krirobo@gmail.com> for reporting the issue.
(Bitbake rev: e008d9bb07e5d1a3584cc04ca2cd3dd906fd5759)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
slef.self is clearly meant to be self, fix typo.
Otavio spotted and reported, thanks.
(Bitbake rev: 316daad7928a58cdfc42e27b20e739f4dd74a02a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
On rare occasions it's useful to be able to exclude a part of a
variable's value from the variable's signature; for example if you want
to add an item to a list sometimes and not have the signature of the
variable change depending on whether the item is in the list or not. The
initial intended use case for this in OpenEmbedded is to allow adding a
function to SSTATEPOSTINSTFUNCS in buildhistory.bbclass and not have
that change any task signatures (so adding and removing
INHERIT += "buildhistory" won't lead to any rebuilds).
Part of the fix for [YOCTO #5897].
(Bitbake rev: f803bf8cfefafcbe212442e66b301ccd9c5aa2a5)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The svk fetcher never appears to be used by anyone and the development
on svk appears to have stopped in 2010. We might as well drop support
for it.
(Bitbake rev: 8239264753977bd06ad5b1b574245d3842af489b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using the "execs" information in new code, it became clear that
the returned data was incorrect and there were missing exec'd functions.
This corrects the error and changes one of the test results to match
the correct behaviour.
(Bitbake rev: 8a24f2d3b735bbc59ca4a09670cabbadb1868c1a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These two functions have little in common, separate them out.
(Bitbake rev: 7413e0fa4dca9571ea98f32dab87d4fd60bc8de9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Start to clean up wget fetcher command construction to allow clearer
and more extensible code structure. Drops support for ${URI} and
${FILE} directly in the commands.
(Bitbake rev: 4e59fe45be2088996abc21e9a631a32b9a9642c9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These variables were dropped from OE-Core some time ago, drop their
usage from the fetcher as well.
(Bitbake rev: bd33e709ab65d6966b234010641861834d170e2b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Clean up some horrible old code and drop usage of the old style *COMMAND
variable and MKTEMPDIRCMD whilst in here. This means we don't need to touch
OVERRIDES either.
(Bitbake rev: c127bb3a9b7b1d2ab2c833ff73186b6ead0dc29c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Clean up some horrible old code and drop usage of the old style *COMMAND
variable and MKTEMPDIRCMD whilst in here. This means we don't need to touch
OVERRIDES either.
(Bitbake rev: 6b79789769da160d0e7fca0f9c6044dc1e11a107)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is other code which can want to run ls-remote style commands with
different parameters so split out the function.
(Bitbake rev: 13f1138f5504feee0ee8e8f3a0675d0bea490351)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use a newly created "git-submodule-test" repo on git.yoctoproject.org
which currently contains one submodule (the bitbake repository).
(Bitbake rev: a750c57242928c546a5aace632543e956ee908eb)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was found when trying to run the data tests individually.
(Bitbake rev: e4bf4ad4b99978483541a719105c98ea124e8a34)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Git versions older than 1.7.10 put absolute paths in configuration files
for the submodule repositories, leading to errors when the repository
checkout is moved. We move the repository as a matter of course in the
gitsm fetcher; the failure occurs in do_unpack). Change the absolute
paths to be relative during processing to fix this.
(At the time of writing, Ubuntu 12.04.4 LTS ships Git version 1.7.9.5,
hence the desire to fix this rather than just mandating a newer Git
version.)
Fixes [YOCTO #5525].
(Bitbake rev: e700d5a41deed4ee837465af526ed30c8a579933)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton
Peter Kjellerstedt
Irina Patru
Richard Purdie
Alexandru DAMIAN
Marius Avram
Cristiana Voicu
Olof Johansson
Valentin Popa