Commit Graph

4340 Commits

Author SHA1 Message Date
Michael Wood 8b3146007f bitbake: toaster: lsupdates Add progress information and clean up logging
Adds basic progress % information and provides better description of
what is happening.

(Bitbake rev: 6393dbf97d450d2521c0bc9429da0987bb7720ec)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:36 +01:00
Michael Wood bb260e94a6 bitbake: toaster: tests: Remove references to LayerSource model
Remove and replace layersource model references in the tests and test
data. Remove the orm/test as this only tested LayerSource interactions
which have now been removed.

(Bitbake rev: 61a47cbc92c856690cb0e8da7102b2e669eaee0a)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:36 +01:00
Michael Wood ef627d0ab8 bitbake: toaster: Replace references to LayerSource models
Replace references to the now deprecated layersource models across
Toaster with the new enums for layer source types.

(Bitbake rev: 48c09c62eb979d840132e58144f0d81ffee675b1)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:36 +01:00
Michael Wood ffc78d329d bitbake: toaster: lsupdates Add layerindex fetcher
Move and refactor the layerindex layer source update mechanism so that
we don't have to track the layerindex objects in the toaster database.
Move this out of the orm and into the management command.

Paves the way for future improvement to allow you to specify a layer
index server as an argument to the command.

(Bitbake rev: f83527edc6d52a34cd73a9c3650ee484407e2e0c)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:36 +01:00
Michael Wood 853450befc bitbake: toaster: models Remove LayerSource models and replace with enum
We had a lot of complexity around different layer sources to allow for
multiple sources and different priorities for these source. This was
implemented using rigged abstract classes which represented the
different layer sources when in fact just an enum/flag on the
layer_version object is sufficient for our needs.

Remove the LayerSourcePriority object as this is not needed. We no longer
have a problem of multiple layers coming from multiple sources so this
is not needed. Two migrations are added to first remove the child models
which represented layersources. Then a second migration is needed to
remove the LayerSource model it's self as Django can't understand the
non-standard base class dependency. Triggering this issue:
https://docs.djangoproject.com/en/1.8/topics/migrations/#dependencies

Clean up a number of flake8 warnings in classes which were modified.

[YOCTO #9853]

(Bitbake rev: 26624740418de95adb1f4a970a30a5f1149ebb79)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:35 +01:00
Michael Wood 20f939f242 bitbake: toaster: bldcollector admin Remove LayerSourceAdmin
Remove the LayerSource admin from django admin interface. LayerSources
are not going to be manageable from the admin interface.

(Bitbake rev: 1c20ffcf88c5b73f6cf5e9b69b81e6bc8cd2493d)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:35 +01:00
Michael Wood 1477886deb bitbake: toaster: loadconf remove Loading LayerSources
We don't need to configure layer sources in the initial configuration as
this information is provided by the models.

(Bitbake rev: fd56c152699bc4c2d22b87728d0fefbff5209135)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:35 +01:00
Paul Eggleton c27ae255db bitbake: lib/bb/progress: avoid possibility of start event being reported twice
In MultiStageProgressReporter, set a guard when we start the progress
so that it can't happen more than once. This fixes "Initialising
tasks.." being shown twice in succession when running bitbake in
non-interactive terminal mode.

(Bitbake rev: 923e68e069127ee7f6e11b91eb1cfa09d502a110)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:35 +01:00
Paul Eggleton 5f4559b2eb bitbake: knotty: don't display ETA for tasks with progress
It turns out that progress information we can extract from a task is
rarely apportioned closely enough to the time taken for the ETA to be
accurate, so showing it is going to be misleading most of the time for
anything but the most basic of examples. Let's just remove it and avoid
misleading (or worse, annoying) the user.

Fixes [YOCTO #9986].

(Bitbake rev: 235db4870b11db97250979e647b54cdb5ce4fbb6)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:35 +01:00
Paul Eggleton 23a551bea1 bitbake: knotty: fix some minor bugs in BBProgress
If you specify custom widgets then we don't want to assume where the
"extra" position is - you should have to specify it, and if it isn't
specified it shouldn't just wipe out the last widget or you can start to
see odd behaviour if you're modifying the code.

(Bitbake rev: 19e33c10feb1637589ceb05b5e8d58b1e012ccb8)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:35 +01:00
Enrico Scholz 0c3ce68410 bitbake: fetch: copy files with -H
When using a PREMIRROR with plain (non-unpack) files, a SRC_URI like

SRC_URI = "file://devmem2.c"

will cause devmem2.c to be a symlink in the WORKDIR pointing to the
local PREMIRROR.

Trying to apply a patch on this file will either modify the file on
the PREMIRROR or will fail due to sanity checks:

ERROR: devmem2-1.0-r7 do_patch: Command Error: 'quilt --quiltrc /cache/build-ubuntu/sysroots/x86_64-oe-linux/etc/quiltrc push' exited with 1  Output:
Applying patch devmem2-fixups-2.patch
File devmem2.c is not a regular file -- refusing to patch

(Bitbake rev: cfd481fe9799e7a4c6bfac32e56cc91cfcd81088)

Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:35 +01:00
Richard Purdie 7f6b6b2ab9 bitbake: cache: Don't interleave pickle cache file writing
For some reason the data written in this way is coming back out the
files out of order. I've not been able to simplify the test case to a
point where this was standalone reproducible. Simplify the code and
write out the cache files sequentially since this seems to avoid the
errors and makes the code more readable.

(Bitbake rev: 14ec47f5f0566dbd280fae8a03160c8500ad3929)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:35 +01:00
Richard Purdie 5d41200113 bitbake: cache: Add better cache loading sanity checks
We've seen cache corruption where the pairs come out in a different
order to the way we saved them for unknown reasons. Add better sanity
checking to give a more user friendly error rather than a crash/traceback.

Also allows the system to reparse and recover.

(Bitbake rev: 4be4a15491530bd6dc018033ad3d4b2562ab6e23)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:35 +01:00
Richard Purdie 40d45cf7a6 bitbake: cache: Drop/simplify pointless type checking
Since we no longer have random data like version fields in these structures
and we can assume any extra cache data subclasses our class, simplify the
code.

This is mostly reindenting after removal of the pointless type checks.

(Bitbake rev: 5eb36278ac9975de1945f6da8161187320d90ba7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:35 +01:00
Richard Purdie c9e65c5d29 bitbake: cache: Improve versions fields handling
Firstly, don't store the versions fields in memory in the cache objects
data store. This just complicates the code for no good reason.

Secondly, write the version fields to all cache files, not just the
core one. This makes everything consistent and easier.

(Bitbake rev: cb666262b2f986b5d9331dfb30458ef1a151fa4d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:35 +01:00
Richard Purdie 09f6a56aa4 bitbake: cache: Correctly handle missing extra caches
If an "extras" cache file is corrupted, the system would not notice
and later fail with errors about missing entries. Add a test for this
which means we can fall back to re-parsing in those cases.

[YOCTO #9902]

(Bitbake rev: 51843d8f2bbe2e54db7593ca61984abe70423ef6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:34 +01:00
Richard Purdie b2ef908be1 bitbake: cache: Move the parsing message to a more logical place
Otherwise you can look at the log and wonder why parsing isn't happening
when it really is due to other code paths.

(Bitbake rev: b48d95677a4d285a77cda2892179965f7f8f06dd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:34 +01:00
Ed Bartosh 317708235b bitbake: bitbake: implement idle timeout for xmlrpc server
Idle timeout can be specified either by -T/--idle-timeout option or
by sessing BBTIMEOUT environment variable. Bitbake xmlrpc server
will unload itself when timeout exprired, i.e. when server is idle
for more than <idle timeout> seconds.

[YOCTO #5534]

(Bitbake rev: 5fa0b3a19e7d0f40790f737abeddf499d53f1f6a)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-21 07:48:52 +01:00
Ed Bartosh 4cba010529 bitbake: bitbake: implement --foreground command line option
This option makes bitbake xmlrpc server to run in foreground.
It should be useful for debugging purposes.

(Bitbake rev: 9d4254be5853a546a346bf0d19919dcfba12773d)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-21 07:48:52 +01:00
Richard Purdie 90436919f8 bitbake: cooker: Fix incorrect dot file generation
In the runqueue cleanup/conversion, "dep" was mistakenly used where "tid" should
be leading to incorrect task-depends.dot files and causing general confusion.
Fix this, its clearly incorrect looking at the code.

(Bitbake rev: 689730dbb068c5ea3593e7b92fe5d5e5c0c3760a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-20 16:46:12 +01:00
Aníbal Limón 1bf1e2c208 bitbake: toaster-tests: add class SeleniumTestCaseBase for browser tests
In order to reuse the Selenium helper outside a Django
environment (for functional testing), add a new module
containing the base class SeleniumTestCaseBase, which only
inherits unittest.TestCase

Add a class SeleniumTestCase with multiple inheritance of
StaticLiveServerTestCase and SeleniumTestCaseBase to prevent
existing tests from breaking.

(Bitbake rev: 1177b3f368d8b7f8557eb649adb2e327b6df801c)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-20 10:38:52 +01:00
Ed Bartosh 63ff759627 bitbake: bitbake: main: implement server autostart feature
If environment variable BBSERVER == 'autostart' bitbake will
automatically load server if it's not running yet.

If host and port are in bitbake.lock then bitbake tries to check
if server is running and responses to commands and starts new
server only if this check fails.

[YOCTO #5534]

(Bitbake rev: 89c6e625d47303b2aad8e6645762f17aee01b2d4)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-20 10:23:46 +01:00
Ed Bartosh b190c08b48 bitbake: bitbake: utils: add BBSERVER to the list of preserved variables
All environment variables that are not in the list returned by
preserved_envvars_exported are cleaned by bb.utils.clean_environment.

Added BBSERVER to the list as we need to access it in bb/main.py
after the call of bb.utils.clean_environment.

(Bitbake rev: 15c4ea679f4fe097a9f21cccfc82907b5f39a4e4)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-20 10:23:46 +01:00
Ed Bartosh 6597ba6b43 bitbake: bitbake: xmlrpc: implement check of connection to server
Implemented check_connection function. The purpose of this function
is to check if bitbake server is accessible and functional.
To check this this function tries to connect to bitbake server and
run getVariable command.

This API is going to be used to implement autoloading of bitbake
server.

[YOCTO #5534]

(Bitbake rev: 1a18f5ceb478f766b53850451549333f655621ea)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-20 10:23:45 +01:00
Ross Burton dead7b22b4 bitbake: lib/bb/utils: no need to unsetenv when manipulating os.environ
Doing both os.unsetenv(foo) and then del os.environ[foo] is pointless as del
will call unsetenv automatically.

(Bitbake rev: a4463e2ff3c7d234320176d671719243292f1af0)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-20 10:23:45 +01:00
Paul Eggleton 45baa90c8c bitbake: fetch2: implement progress support
Implement progress reporting support specifically for the fetchers. For
fetch tasks we don't necessarily know which fetcher will be used (we
might initially be fetching a git:// URI, but if we instead download a
mirror tarball we may fetch that over http using wget). These programs
also have different abilities as far as reporting progress goes (e.g.
wget gives us percentage complete and rate, git gives this some of the
time depending on what stage it's at). Additionally we filter out the
progress output before it makes it to the logs, in order to prevent the
logs filling up with junk.

At the moment this is only implemented for the wget and git fetchers
since they are the most commonly used (and svn doesn't seem to support
any kind of progress output, at least not without doing a relatively
expensive remote file listing first).

Line changes such as the ones you get in git's output as it progresses
don't make it to the log files, you only get the final state of the line
so the logs aren't filled with progress information that's useless after
the fact.

Part of the implementation for [YOCTO #5383].

(Bitbake rev: 4027649f422ee64b1c4e1ad8d48ac295050afbff)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 15:04:37 +01:00
Elliot Smith 8d08a73225 bitbake: toaster-tests: package count/size shouldn't show for non-image builds
If a build doesn't produce any image files, the package count
and size shouldn't be shown.

Also add some metadata to build dashboard elements so it is clear
what they're for, and so they can be queried by the tests.

(Bitbake rev: 3d5090af4475b1d0bb56911a8e30abf9097c1b3c)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:52 +01:00
Elliot Smith 8ccf8392aa bitbake: toaster-tests: new custom image button shouldn't show for non-image builds
[YOCTO #9514]

(Bitbake rev: 9bc014d89434400c1493fa9f07ce3a51d37dab51)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:52 +01:00
Elliot Smith db5426b079 bitbake: toaster-tests: add tests for build artifact display on build dashboard
Add tests for display of image, kernel and SDK artifacts on the
build dashboard, checking that the "Images" option in the left-hand
menu and the "Build artifacts" section display correctly for
different types of build.

Also add metadata to elements on the build dashboard so it's clearer
what they represent, and to assist in finding them in the tests.

Add a method to the test helper to make it more convenient to check
whether a single element matching a selector exists.

[YOCTO #8556]
[YOCTO #8563]
[YOCTO #9500]

(Bitbake rev: 644a888ce5a2141f2e6e1c22430e196b65cb1313)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:52 +01:00
Elliot Smith 9475a684c4 bitbake: toaster: the customise image button shouldn't rely on targets
The build dashboard customise image button (for creating a new
custom image based on an image recipe used by a build) shouldn't
rely on targets: whether a new custom image can be created or not
depends on whether any of the recipes used by the build are image
recipes.

Modify the method used to determine whether a build has customisable
images to look at the image recipes used during the build, rather
than whether the targets run by the build refer to image recipes.

(Bitbake rev: 6648876c91134bda8498b4f8d7ace9147ec0d985)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:52 +01:00
Elliot Smith 2db40e3771 bitbake: toaster: add package manifest path to Target objects
Store the path to the *.rootfs.manifest file for targets which
generate images.

A link to the package manifest is displayed in the build dashboard
for targets which produce image files.

Like the license manifest path, if a target would have produced
the package manifest (but didn't, because it already existed), that
path is copied from the target which did produce the package
manifest.

(Bitbake rev: 79b8e349a0da2ea6b97ad82daa5837e6dfffe0af)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:52 +01:00
Elliot Smith 1027e0e313 bitbake: toaster: better display of targets which produced no images
SDK targets (populate_sdk) produce SDK artifacts but no image files.
Currently, these targets appear under the "Images" heading in the
build dashboard, even though they aren't strictly image targets.

Change the heading to "Build artifacts". Also remove the section
which states that a build produced no image files: this is not
correct for populate_sdk targets (those targets don't produce
image files under any circumstances); and other changes mean
that all targets which do produce images will now show those
files.

The check for whether to display the "Build artifacts" section also
needs to change, as we show targets here which didn't produce any
images but did produce SDK artifacts.

[YOCTO #8556]

(Bitbake rev: b4dce68045c4615e7a6a474e952f670721a3b54e)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:52 +01:00
Elliot Smith adbf206526 bitbake: toaster: show manifests in their own section of the build dashboard
In the build dashboard, add a section for manifests to each target,
which will contain the license and package manifests for a target.

As we don't record the package manifest (yet), just move the
license manifest link to that section for now.

(Bitbake rev: 58cb685b567a081834a605530957f165d35fcf5e)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:52 +01:00
Elliot Smith 07a58a8944 bitbake: toaster: only show "New custom image" button for builds with image targets
Add a has_image_targets() method to Build, and use that to hide
the "New custom image" button on the build dashboard if a build
has no targets which build images.

[YOCTO #9514]

(Bitbake rev: 3c4b053e44ea512ef2ced67289a7b0161db6ce9b)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:52 +01:00
Elliot Smith 587275eefd bitbake: toaster: use has_images() methods to display images correctly
In the build dashboard, we had issues with showing images correctly,
as we were using the is_image property of targets to determine
whether a target would have image files. This property can
be set to True if a target refers to an image recipe
(e.g. "core-image-minimal"), even if the task used in the build
didn't produce any image files.

By adding has_images() methods to the Target and Build objects,
which count associated Target_Image_File objects,
we can correctly determine whether a target has image files
associated with it, and if any of the targets for a build has
image files. This means that we can screen out the left-hand
"Images" menu options for builds which contained image-related
targets (e.g. "core-image-minimal") but which didn't produce
any images (e.g. "rootfs" task).

[YOCTO #9500]
[YOCTO #9784]

(Bitbake rev: f6bba0ff254d5ed3163151d4b938f3a43c9acb0a)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00
Elliot Smith 150e5588a0 bitbake: buildinfohelper: only record image files for tasks which make images
If a target is built which is classified as an "image" target
(e.g. "core-image-minimal"), Toaster reads the list of files in
the image (from the files-in-image.txt file).

However, Toaster continues to do this for builds which don't
produce images, if the recipe providing the target is an
image recipe. This can result in a list of files in the image
being attached to a target which didn't produce an image (e.g.
rootfs).

When associating files with an image, ensure that only targets
with a task which produces an image have "files in the image"
associated with them.

[YOCTO #9784]

(Bitbake rev: 44375d0c2a88e0070b8067c9285b89c54eaf3152)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00
Elliot Smith 00c2c0be5e bitbake: toaster: improve scan for SDK artifacts
SDK artifacts were previously picked up by toaster.bbclass and
notified to buildinfohelper (via toasterui). The artifacts
were then added to the Build object, so that it wasn't clear
which artifact went with which target; we were also unable
to attach SDK artifacts to a Build if they had already been
attached to a previous build.

Now, toaster.bbclass just notifies the TOOLCHAIN_OUTPUTNAME when
a populate_sdk* target completes. The scan is moved to buildinfohelper,
where we search the SDK deploy directory for files matching
TOOLCHAIN_OUTPUTNAME and attach them to targets (not builds).

If an SDK file is not produced by a target, we now look for a
similar, previously-run target which did produce artifacts.
If there is one, we clone the SDK artifacts from that target
onto the current one.

This all means that we can show SDK artifacts by target, and should
always get artifacts associated with a target, regardless of whether
it really build them.

This requires an additional model, TargetSDKFile, which tracks
the size and path of SDK artifact files with respect to Target
objects.

[YOCTO #8556]

(Bitbake rev: 5e650c611605507e1e0d1588cd5eb6535c2d34fc)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00
Elliot Smith f39ae146ea bitbake: buildinfohelper: fix retrieval of targets
When buildinfohelper records the targets for a build, it looks
up any existing targets for a build and creates them if they
are not present. This is because in the case of Toaster-triggered
builds, the Target objects have already been created (inside
triggerBuild()) and don't need to be recreated; but in the case
of cli builds, the Target objects have to be created by
buildinfohelper.

The issue is that the code for retrieving an existing target for
a build only looks for Targets with a matching target and build,
e.g. Targets for build X with target "core-image-minimal". But it
is perfectly legitimate to call bitbake with a command like
"bitbake core-image-minimal:do_populate_sdk
core-image-minimal:do_populate_sdk_ext". In such a case, the
code which looks for matching targets finds two objects, as it
doesn't filter by task.

Add the task into the filter for the Target so that only one
Target object is be returned. Note that a command
line like "bitbake recipe:task recipe:task" will still cause an
error as bitbake doesn't de-duplicate the command line arguments
and will run the recipe:task combination twice.

(Bitbake rev: 1c0a689fdaae6469d4afb98583161073d32ea50b)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00
Elliot Smith 4125da7763 bitbake: toaster: attach kernel artifacts to targets
The bzImage and modules files were previously attached to a build,
rather than to the target which produced them. This meant it was
not possible to determine which kernel artifact produced by a
build came from which target; which in turn made it difficult to
associate existing kernel artifact with targets when those
targets didn't produce artifacts (e.g. if the same machine + target
combination was built again and didn't produce a bzImage or modules
file because those files already existed).

By associating kernel artifacts with the target (via a new
TargetArtifactFile model), we make it possible to find all
the artifacts for a given machine + target combination. Then, in
cases where a build is completed but its targets don't produce
any artifacts, we can find a previous Target object with the same
machine + target and copy its artifacts to the targets for a
just-completed build.

Note that this doesn't cover SDK artifacts yet, which are still
retrieved in toaster.bbclass and show up as "Other artifacts",
lumped together for the whole build rather than by target.

[YOCTO #8556]

(Bitbake rev: 9b151416e428c2565a27d89116439f9a8d578e3d)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00
Elliot Smith e9808576da bitbake: toaster: improve image file suffix retrieval
Refactor retrieval of suffix from image file path, making it a
a method on Target_Image_File. This makes it easier to use this
in the build dashboard for individual images, plus reduces the
complexity of the code required to get all of the image file
suffixes for a build.

(Bitbake rev: 9c38de3dec74c122c2060cad37331bdafc6858ec)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00
Elliot Smith 5dfa120a7c bitbake: toaster: do image and artifact scan on BuildCompleted
Move the image and artifact scan code from toaster.bbclass and
consolidate its logic with the existing logic in buildinfohelper.

Remove handler setup for events which used to be fired from
toaster.bbclass but which are now handled directly by buildinfohelper.

[YOCTO #8556]

(Bitbake rev: f0085cd554604cfff4a3f40a34825fbb6878004f)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00
Elliot Smith b0585e6b0c bitbake: toaster: display Target targets in build dashboard
The build dashboard was showing the targets for the build in the page
heading and title as "Target object".

Add a filter which extracts the "target" from each Target object
as a string so that the heading and title display correctly.

Also sort the image file suffixes alphabetically.

[YOCTO #8556]

(Bitbake rev: 33a24992139e7dc05d14bbe1da60299b732945aa)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00
Ed Bartosh 844e1f9e1d bitbake: runqueue: improve exception logging
Runqueue errors direct the user to view the "failure below",
but no additional error message is available.

Log the stacktrace so that the user can see what went wrong.

Also fix a typo in the log message.

(Bitbake rev: e191f401e372ee181bc02250232ad9cb9a0e9477)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00
Robert Yang 7a220aeaae bitbake: lib/bb/utils.py: return sorted OrderedDict in explode_dep_versions2
The OrderedDict's item is sorted by insertion order, there might be a
problem when build the same recipe again, for example:
- First build of acl:
  Depends: libattr1 (>= 2.4.47), libc6 (>= 2.24)
- Second build of acl:
  Depends: libc6 (>= 2.24), libattr1 (>= 2.4.47)

They are exactly the same depends, but tools like "diff" doesn't think
so. Return sorted OrderedDict will fix the problem.

(Bitbake rev: a392f19f16ef8202ce3c12afbeb186a02438da17)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00
Paul Eggleton bc3a206de3 bitbake: knotty: avoid errors when fetching outside of a task
In a few places we use the fetcher code to fetch files outside of a
task, for example uninative in OE. In that case the pid of the event is
0 and that was causing an error in BBUIHelper.eventHandler(). Check the
pid and do nothing if it's 0.

(Bitbake rev: 59cb919e5cd5c653fb4d69b2d6a4320648443e10)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:50 +01:00
Ross Burton 2afcbfef2c bitbake: build: don't use $B as the default cwd for functions
When bitbake executes a shell or Python function it can cd/chdir() into a
directory before executing the task. If no directory is specified then the
default of $B is used.  However $B is an OpenEmbedded variable and BitBake
shouldn't be aware of it.

To solve this change the semantics slightly so that if no directory is
specified, the current working directory isn't changed.  There's also a sanity
check that emits a warning if a Python task does os.chdir() without restoring
the old path, and the previous working directory is restored.

This does change semantics: whereas before a function in OE would have $B as the
working directory unless specified, now the working directory is the top of the
build tree.  Any breakage this causes can be solved by either adding
do_some_task[dirs] = "${B}" or by using absolute paths in the task.

[ YOCTO #4634 ]

(Bitbake rev: 67a7b8b021badc17d8fdf447c250e79d291e75f7)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:50 +01:00
Ross Burton faa726824d bitbake: lib/bb/build.py: decode the command as UTF-8
The messaging FIFO is UTF-8, so decode the command as UTF-8 as well as the value
as otherwise "bberror" != b("bberror") and none of the messages from shell
functions are ever displayed.

Also add an else to the command parser so unhandled commands are noticed.

[ YOCTO #9947 ]

(Bitbake rev: 42d727743fa599e0a3c5ad2c29a1e6ede1a918bb)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:47:06 +01:00
Ross Burton b7a741cbca bitbake: lib/bb/utils: fix set_process_name
With Python 3 create_string_buffer needs a bytes() not a str() but as we were
catching all exceptions nobody noticed.

[ YOCTO #9910 ]

(Bitbake rev: 6576a9a95486c28a01d4211b4a33cc3e2c55a7cc)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:47:06 +01:00
Michael Wood 8dc159dd93 bitbake: toaster: tests Add selenium test for layerdetails page
This tests:
 - Adding remove layer from project
 - Deleting layer
 - Editing layer fields

(Bitbake rev: cabe7e2459fcd561bced2d39ba5bd173576153e5)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:29 +01:00
Michael Wood 8813726f1d bitbake: toaster: add Layer delete front end feature to layerdetails
Add the front end feature to delete a layer from the layer details page.

[YOCO #9184]

(Bitbake rev: 91815229f60eb9deba7d299f05c69b52ff1df59c)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:29 +01:00