Commit Graph

4645 Commits

Author SHA1 Message Date
Michael Wood 3025092142 bitbake: toaster: orm Add a constant for the CustomImageRecipe's layer name
Use a constant to define the name for the toaster custom images layer;
this constant is then used to identify this layer in various places.

(Bitbake rev: 2540969ec71612af7f9041cadcc401513e9b357b)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-23 17:53:55 +00:00
Michael Wood 3df6551d8c bitbake: toaster: localhostbecontroller Don't clear out toaster custom layer dir
We may have a recipe which is based on a custom image recipe that has
already been built so keep the recipe file around so that it can be read
by the generate recipe function.

(Bitbake rev: 502148fe498fe0a41ae89c9f649d3cb1253b0487)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-23 17:53:55 +00:00
Richard Purdie 8037ba4d86 bitbake: bb/tests/fetch: Update cups url
Update the upstream url used for testing cups versions after upstream website
changes.

(Bitbake rev: 5f06041d4936fc22297945bbbad7020bfa9083c6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-21 12:43:12 +00:00
Brendan Le Foll 3c66619d50 bitbake: fetch2/npm: fix ud.registry so that alternative registries can be handled
Fixes [YOCTO #9231]

npm when given an invalid registry URL with --registry actually goes and
fetches from the default registry, but this commit makes sure it goes to the
specified one.

(Bitbake rev: 7c849be7c70a5db4f66fe3041486abb923b5e4ee)

Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-12 09:35:52 +00:00
Lucas Dutra Nunes d53413d3a8 bitbake: server/process: Try connecting 4 times before giving up
Instead of trying one time with a timeout of 20 seconds try 4 times with
a timeout of 5 seconds, to account for a slow server start.

(Bitbake rev: 4a7fe63126dd8177baa5ad21e59e0bebeea8c596)

Signed-off-by: Lucas Dutra Nunes <ldnunes@ossystems.com.br>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 22:47:29 +00:00
Michael Wood 0f0105932c bitbake: toaster: models List only have the specified project's imported layers
When returning the compatible layers make sure that we are only
listing: All the layers which are for this release && configuration
layers (i.e. aren't part of the build history) and which aren't an
imported layer OR are this project's imported layer(s).

[YOCTO #8944]

(Bitbake rev: de8baedaccb451c12bc3f642449db3f64aed6bf7)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 22:47:29 +00:00
Elliot Smith 0dcab0258e bitbake: toaster: rework task buildstats storage and display
The data available from buildstats is now more fine grained than
previously, so take advantage of that to enrich the data we save
against tasks:

* Store the CPU usage for user and system separately, and display
them separately.
* Disk IO is now measured in bytes, not ms. Also store the
read/write bytes separately.
* Store started and ended times, as well as elapsed_time. This
will enable future features such as showing which tasks were
running at a particular point in the build.

There was also a problem with how we were looking up the Task
object, which meant that the buildstats were being added to
new tasks which weren't correctly associated with the build. Fix
how we look up the Task (only looking for tasks which match the
build, and the task and recipe names in the build stats data) so
the build stats are associated with the correct task.

[YOCTO #8842]

(Bitbake rev: efa6f915566b979bdbad233ae195b413cef1b8da)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 22:45:16 +00:00
Sujith H cc74a8ae26 bitbake: toaster: use force_bytes to display non-ascii project names
When user enters a non-ascii character in the project
name of toaster, the build doesn't get triggered.
Use force_bytes to fix this.

Also deal with non-ascii project names when logging the
build request in runbuilds.

[YOCTO #9071]

(Bitbake rev: b6141c4d170885d3bdf63074afcb1e41fde0a8f0)

Signed-off-by: Sujith H <sujith.h@gmail.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 22:45:16 +00:00
Peter Kjellerstedt aebc22dbfa bitbake: fetch2: Make SRC_URI[md5sum] and SRC_URI[sha256sum] expand their values
For some reason, the values for SRC_URI[md5sum] and SRC_URI[sha256sum]
were not being expanded. That lead to the following code not working
as expected:

SRC_URI = "http://.../${PN}-${PV}.tar.gz"

MD5SUM = "123abc..."
SHA256SUM = "abcd1234..."

SRC_URI[md5sum] = "${MD5SUM}"
SRC_URI[sha256sum] = "${SHA256SUM}"

(Bitbake rev: ba011470df0ea8bd89f01c0b02ec4b3969e60ce7)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 22:45:16 +00:00
Ed Bartosh d405f97af3 bitbake: xmlrpc: fix bug in setting XMLRPCServer.single_use
XMLRPCServer.single_use attribute was always set to False.
This caused xmlrpc server to keep running after build is done as
BitBakeServerCommands.removeClient only shuts down server if its
single_use attribute is set to True.

(Bitbake rev: 0a60b0928a0a746a60d2c2f294ff1903963c7086)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 22:45:16 +00:00
Paul Eggleton c50bdb300b bitbake: fetch2/npm: add missing URL argument to ParameterError
Without this you get a rather odd traceback instead of the proper
exception message.

(Bitbake rev: 2fe1826d3077eeda6cde433d3a1e6620f74e08dd)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 22:45:16 +00:00
Paul Eggleton fbf27c4c5d bitbake: fetch2/npm: properly handle npm dependencies
The output of "npm view dependencies" isn't entirely JSON if there are
multiple results, but the code here was just discarding the output if
the entire thing didn't parse as JSON. Split the output into lines and
iterate over it, parsing JSON fragments as we find them; this way we end
up with the last package's dependencies since it'll be last in the
output.

Digging further, it seems that the dependencies field reported by "npm
view" also includes optional dependencies. That wouldn't be a problem
except some of these optional dependencies may be OS-specific; for
example the "chokidar" module has "fsevents" in its optional
dependencies, but fsevents only works on MacOS X (and is only needed
there). If we erroneously pull in fsevents, not only is it unnecessary
but it causes "npm shrinkwrap" to throw a tantrum. In the absence of a
better approach, look at the os field and discard the module (along with
any of its dependencies) if it isn't for Linux.

As part of this, we can reduce the calls to npm view to one per package
since we get the entire json output rather than querying twice for two
separate fields. Overall the time taken has probably increased since we
are being more thorough about dependencies, but it's not quite as bad as
it could have been.

(Bitbake rev: 436d67fe7af89ecfbd11749a6ae1bc20e81f2cc8)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 22:45:16 +00:00
Paul Eggleton ef6a4518e4 bitbake: fetch2/npm: fix errors with some version specifications
"2 || 3" is a valid version specification for a dependency in an npm
package.json file, but of course that looks like something else when
sent to a shell. Quote the version value to avoid this.

(Bitbake rev: bea0246831a46d943d2e27d6b38f6e498bd3413c)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 22:45:16 +00:00
Ed Bartosh 3322fa7670 bitbake: toasterui: fix warning 'Unknown event'
Continue after processing BuildStarted event to fix
WARNING: Unknown event: <bb.event.BuildStarted object at 0x2554150>

(Bitbake rev: 12f1fb8c9b70fea0c9145f881bcceb8af32df6af)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:04 +00:00
Ed Bartosh 621cbc86c9 bitbake: toasterui: exit on final events
Toasterui exits only if bitbake observer shuts down.
In build mode it should exit when build is done.

Made toasterui exit on bb.command.CommandCompleted,
bb.command.CommandFailed and bb.command.CommandExit events
when it's running in build mode.

(Bitbake rev: b11f9d6d3c2eb615335901e1dcea699daf3afb4c)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:04 +00:00
Ed Bartosh 8e138b7f2a bitbake: toasterui: make toasterui to work in build mode
Currently toasterui works only in observer mode. This is
artificial limitation which was made to support current toaster
design. As we decided to stop using bitbake server we'll
need to run toasterui also in build mode.

[YOCTO #7880]

(Bitbake rev: d4b5796899c3ca5c7becd7322291afd8afb35a31)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:04 +00:00
Ed Bartosh 0a6130697e bitbake: toasterui: check if setEventMask succeeded
Currently toasterui ignores return value of setEventMask
command, which created confusing difference between set of
events set by this command and the real set used in the code.

Checked if setEventMask succeeded. Print error message and
exit if it's not.

(Bitbake rev: 6e3f13ffb47102b5df2da91fbc3f5da3179245b2)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:04 +00:00
Ed Bartosh ac941ac0a2 bitbake: command: make setEventMask readonly
Executing setEventMask command when bitbake server is in readonly
mode causes runCommand to fail with the following error:
 'Not able to execute not readonly commands in readonly mode'

Set readonly attribute for setEventMask command to make it working
for Toaster UI. This should not do any harm as this command doesn't
influence cooker state.

(Bitbake rev: 8a47d30b2555255fbf6049c5ed69b29664c32b17)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:04 +00:00
Ed Bartosh dd3da9aa47 bitbake: toasterui: update list of events
Removed events not used in the code from the list.
Added events that are used in the code.

(Bitbake rev: 16b14ec16049cc2040a60ad5fc95f6e19dda91a6)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:04 +00:00
Ed Bartosh f56fa5dfd4 bitbake: toasterui: reformat list of events
Reformatted and reordered list of events to make changes
easily and see them clearly in the diffs.

(Bitbake rev: 42a2d1115f2b23dc063a3172285ca3be73cf70bb)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:04 +00:00
Ed Bartosh a71d32ac53 bitbake: toaster: remove sshbecontroller module
The code of this module is broken for a long time.
The functionality of it can be easily achieved by running
'manage.py runbuilds.py' on remote machine.

[YOCTO #8806]

(Bitbake rev: 975081eefdd7041a6b4bef6842c1bac9799a1b44)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:03 +00:00
Ed Bartosh 3db71b4087 bitbake: toaster: don't use sshbecontroller
Removed usage of sshbecontroller from bbcontroller, models, tests
and database schema.

(Bitbake rev: 3ee06eb7e96de5dba539ad52201867e77d06a53e)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:03 +00:00
Ed Bartosh 790b2d1387 bitbake: toaster: raise NotImplementedError
Raised NotImplementedError instead of Exception to be
able to catch it.

This is a preparation for removing sshbecontroller module.
It has to be done as code in bldcontrol/tests.py imports custom
NotImplementedException from sshbecontroller.

(Bitbake rev: c243ab6c83fe12d84777e4c3a18fd393827b9327)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:03 +00:00
Belen Barros Pena 96535ba720 bitbake: toaster: bring back the strict directive
This patch e0fd96442a removed the
"use strict" directive from the projectpage.js file by mistake.

This patch makes amends.

(Bitbake rev: b8044ce60af3f0b064cfba76c577503cc896e358)

Signed-off-by: Belen Barros Pena <belen.barros.pena@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:03 +00:00
Belen Barros Pena 5b8b3993e4 bitbake: toaster: change 'revision' to 'Git revision'
I've received some feedback on the 'Revision' label we use in the import
layer page. It is not quite communicating that what's required is a Git
revision. Changing it to 'Git revision' to make it a bit more specific.

The change applies not only to the import layer page, but to all pages
showing revision information in the project configuration section.

For more on the feedback received, check

https://bugzilla.yoctoproject.org/show_bug.cgi?id=8429#c3

(Bitbake rev: 09392f36a4f115c2432302125e8cac48a9aa304f)

Signed-off-by: Belen Barros Pena <belen.barros.pena@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:03 +00:00
Michael Wood 07ead9869a bitbake: toaster: views api Package info return both kinds of RDEPENDS
As we do with the popover snippet which shows dependencies inline in the
table also show dependencies which have both TYPE_TRDEPENDS and
TYPE_RDEPENDS.

Also remove obsolete comment

(Bitbake rev: d3b5f3b7ba4550e7cd03a37ca19ccd2fc0042b2d)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:03 +00:00
Dave Lerner 9cda2ab8d5 bitbake: toaster: fixup dependency excludes for customimage
[YOCTO #9156]

For a customized image when adding a dependent package X that depends on
dependency package Y, in addition to adding X to appends_set and Y to
includes_set, make sure that Y is no longer in the excludes_set. Y may
have been added to the excludes_set by a prior package removal.

(Bitbake rev: 6b29d3297de2ae48a3ac5529ba0d22f895276b56)

Signed-off-by: Dave Lerner <dave.lerner@windriver.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:03 +00:00
Paul Eggleton a54cebe929 bitbake: fetch2/npm: ignore unknown headers in tarballs
Tarballs that are fetched down via npm repositories seem to often have
unknown headers. This doesn't affect our ability to extract the contents
though so we don't really care to see those warnings.

(Bitbake rev: b38975103e52a0c25e9ad9032c8cca1c47cbdcc2)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:03 +00:00
Paul Eggleton 0cd1be1f09 bitbake: fetch2/npm: handle alternative dependency syntax
npm allows you to specify a dependency as a Github username+path, or
omit the version entirely. You can hit these if you don't use a
shrinkwrap file, with the result that the code later fails due to the
output of "npm view" being empty; so handle this lazily by just ignoring
this part of the dependency if it's not really a version.

(Bitbake rev: 7b7a65c44dbdd5ba9366d4e2093f76df8758d546)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:03 +00:00
Paul Eggleton d9999279d9 bitbake: fetch2/npm: fix indentation
No code changes, just fix to use four spaces.

(Bitbake rev: 66a9ee7d54ca9c25209f72da079f260ccdcc872a)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:02 +00:00
Scott Rifenbark 353b755177 bitbake: bitbake-user-manual: Added expand() function to list.
Fixes [YOCTO #9147]

Added the d.expand(expression) fuction to the list of functions
in the "Accessing Datastore Variables Using Python" section.

(Bitbake rev: 19507b80f35d37dc4b1614bd390b8e261dd4a2bd)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-03 17:34:14 +00:00
Scott Rifenbark 638ad177aa bitbake: bitbake-user-manual: Added note for Python variable ref expansion.
Fixes [YOCTO #9148]

Added a note about Variable expressions (e.g. ${X}) are no longer
expanded within Python functions.

(Bitbake rev: 4ebe55cb8c8db1c5cf5d127e213487c5a453a68a)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-03 17:34:14 +00:00
Scott Rifenbark da22add33b bitbake: bitbake-user-manual: Enhance environment variable discussion.
Fixes [YOCTO #8567]

Updated the variable list describing the variables that affect
how environment variables are handled.  Also updated the section
on how those variables are passed.

(Bitbake rev: 91cb52b1e77bba9d046239933b5c0513d01e6824)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-03 17:34:14 +00:00
brian avery 937ecd07d3 bitbake: toaster: cleanup of bin/toaster startup code
This cleanup fixes a few issues:

    1) Removes the superflous code to set toasterconf.json
       - this isn't used and referenced meta-yocto
    2) Changes exit to return so we don't surprise the user by exiting their shell
       - this is necessary because it is being sourced
    3) Removes the last references to the old TOASTER_MANAGED variable
       - this is historical and no longer used.
    4) Adds -t parameter to lsof
      - This stops it from dying on odd filesystems and is much
        faster since all we are using are processes anyway
    5) Handles start and stop as params
      - it was easy to confuse the script especially
        if we were calling it with parameters.
      - if start/stop isn't specified, it will still toggle

(Bitbake rev: 88fddbe80f56828026bf93560037af52b5dab628)

Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-02 22:41:23 +00:00
brian avery a7d1b95c96 bitbake: ui: remove the puccho ui
This ui does not work in master, nor has it been updated for several years.

[YOCTO #9178]

(Bitbake rev: 9fad1d13eed1f725971e6d12d3977cd31e07019a)

Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-02 22:41:23 +00:00
bavery a9dc72f08b bitbake: hob: removal of hob ui and associated ui files
We've been gearing up the Toaster web UI to replace the Hob (GTK+ based) UI
for some time now; Hob has basically been on life support for the past few
releases. As of late last month in master, Toaster has the capability to
select the packages in an image, removing the last thing that Hob could do
that Toaster couldn't.

To recap, the reasons why Hob is being removed include:

- The code is tightly woven into BitBake, making it fragile. This means it
needs significant QA and maintenance on an ongoing basis.

- Some of the implementation is not ideal; we'll be able to remove some cruft
from BitBake and OE-Core at the same time.

- It's GTK+ 2 based, not the current GTK+ 3.

- Toaster is now a much more capable UI and is being actively maintained

The discussion about removing hob can be found at:
http://lists.openembedded.org/pipermail/openembedded-architecture/2016-February/000082.html

(Bitbake rev: be2cceea159c6ca9111eff3df87b98513eab6d72)

Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-02 22:41:23 +00:00
Brendan Le Foll 27468dbc3a bitbake: fetch2/npm: Add missing ParameterError import
(Bitbake rev: b583a40c9086b3587065995d8cee45b7ea36c1b3)

Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-02 22:41:22 +00:00
Brendan Le Foll 44e3461969 bitbake: npm: in cases where shrinkwrap resolved a git URL, ignore it and grab dist.tarball
npm-shrinkwrap will sometimes resolve a git URL which instead of a http url, in
this case go and grab the dist.tarball via npm instead of using the resolved
URL.

(Bitbake rev: eb53b927ff59aa19cf28bc46beb9f9a185a59990)

Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-02 22:41:22 +00:00
Richard Purdie 2a7318133b bitbake: fetch2: Fix unpack for absolute file urls
The previous commit breaks absolute pathnames in file:// urls, this
fixes it.

(Bitbake rev: b8113a1800687a37a26ac28deafdbafd74cc138e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-02 22:41:22 +00:00
Alexander Shashkevich 865d2feff6 bitbake: fetch2: fixes copying of file://dir; subdir=foo, bug 6128 and bug 6129
When in SRC_URI appears file://dir;subdir=foo unpacker copies 'dir' to ${WORKDIR}, not
${WORKDIR}/foo as it should be.

These changes are fixing following bugs as well:
Bug 6128 - Incorrect wildcard unpack behaviour in fetcher
Bug 6129 - Local directories unpack to a different location than local files

(Bitbake rev: e659a3b0c2771679057ee3e13cd42e6c62383ff2)

Signed-off-by: Alexander Shashkevich <alex@stunpix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-02 22:41:22 +00:00
Richard Purdie 2e27c4bf23 bitbake: fetch2/npm: Enable fetcher
For some reason the enablement piece of the patch went missing, add it.

(Bitbake rev: 0270b5a3873ed0aeca3a66198c87a6164fb644b8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 12:01:07 +00:00
Richard Purdie 030d920e2d bitbake: providers: Fix PREFERRED_VERSION lookup for '_' in PN
PN can contain '_', e.g. gcc-cross-x86_64 and an override cannot
hence we do this manually rather than use OVERRIDES.

(Bitbake rev: 7a6baf02617d1edced4eaff235e73d746e2a3b68)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:34:38 +00:00
Richard Purdie c679a3dd41 bitbake: fetch2: Skip lockfiles and donestamps for local files
For local files, there are no races with downloads, we don't need ".done"
stamps and we don't need lockfiles.

This considerably cleans up DL_DIR and all the pointless ".done" files
as well as removes stalls over local files with the same name.

(Bitbake rev: 48e903745db578d9b9b425a8d411c1369df0eb94)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:34:38 +00:00
Richard Purdie d01042eda9 bitbake: fetch2/__init__.py: Error if lockfile path invalid
Rather than create ".lock" and ".done" files with no name, error,
forcing us to fix the cases where this is a problem.

(Bitbake rev: 81158071508cc68c39db7d501370872f44d335cc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:34:38 +00:00
Richard Purdie ab7b7bf94d bitbake: fetch2/__init__: Fix decodeurl to better handle urls without paths
If we specify urls such as npm://somehost;someparams the fetcher currently
does a poor job of handling mirrors of these urls due to deficiencies in the
way decodeurl works. This is because "somehost" is returned as a path, not
a host.

This tweaks the code so that unless its a file url, the host is returned
correctly.

This patch also adds test cases for these urls to the exist set of test
urls.

We need to tweak the URI() class since this thinks this is a relative url
which is clearly isn't. We also need to handle the fact that encodeurl will
error if passed a url of this form (it would want the path to be '/'.

(Bitbake rev: 83203cd2e677706e0111892a7843b83263cb8bd9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:34:38 +00:00
Richard Purdie 06b4d8ffe4 bitbake: fetch2/wget: Set localfile for directories
If we wget something which looks directory like we end up with lock files
and done stamps without names, they also all use the same lockfile.

This change ensures that we use separate lock files based on the url
and avoid creating the mysterious ${DL_DIR}/.done files.

(Bitbake rev: 20bc82086018832e047345a672d74b6c1c113650)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:34:38 +00:00
Belen Barros Pena e0fd96442a bitbake: toaster: change 'delete layer' to 'remove layer'
I have received quite a few complaints about the use of the word
'delete' for layer removal, so change it to 'remove'. That also matches
the language we use for packages in image customisation.

[YOCTO #9165]

(Bitbake rev: 3c5ac2ddfb3f5ecd3f3218de0d6564e5f3842b98)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:38 +00:00
Belen Barros Pena 6e8282063f bitbake: toaster: rename 'run again' button
I quite dislike the 'Run again' label we use in the button that rebuilds
things. Changing it to 'Rebuild', which is shorter and more specific.

(Bitbake rev: 865a2015e86a1bc5cc7d63308f27c292d1ca98eb)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:36 +00:00
Dave Lerner c8dd72c797 bitbake: toaster: fix banner after customimage package add
Correct formatting of the banner message after adding a package to a
custom image.

[YOCTO #9101]

(Bitbake rev: da233005eb8cfa7842cd1a768c16e42aaaa55fad)

Signed-off-by: Dave Lerner <dave.lerner@windriver.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:35 +00:00
Belen Barros Pena 149f57470c bitbake: toaster: custom breadcrumb for the default project
The default "Command line builds" project does not have a Configuration
page. It therefore needs a custom breadcrumb where the project name goes
to the project builds page, instead of the project configuration page.

(Bitbake rev: 5545acf6703a25ee46776138bbbd804615add89c)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:33 +00:00
Haris Okanovic 4a12865c7b bitbake: prserv: Add dump_db()
Returns a script (string) that reconstructs the state of the
entire database at the time this function is called. The script
language is defined by the backing database engine, which is a
function of server configuration.
Returns None if the database engine does not support dumping to
script or if some other error is encountered in processing.

The SQLite3 implementation in db.py calls iterdump() [1] to generate
a script. iterdump() is the library equivalent of the `sqlite3 .dump`
shell command, and the scripts are compatible. Execute the script in
an empty SQLite3 database using the sqlite3 utility to restore a backup
of prserv.

Use case: Backup a live PR server database in a non-racy way, such
that one could snapshot the entire database after a set of bitbake
builds all using a shared server. I.e. All changes made prior to
the start of a dump_db() operation should be committed and captured
in the script. Subsequent changes made during the backup process are
not guaranteed to be captured.

Testing: ~7MB database backs up in ~1s while PR server is under load
from 32 thread bitbake builds on two separate machines.

[1] https://docs.python.org/2/library/sqlite3.html#sqlite3.Connection.iterdump

(Bitbake rev: 004003daf6bd0f0233ce5c2d95f1d7d64ab91bb3)

Signed-off-by: Haris Okanovic <haris.okanovic@ni.com>
Reviewed-by: Ken Sharp <ken.sharp@ni.com>
Reviewed-by: Bill Pittman <bill.pittman@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:31 +00:00
Dave Lerner bdb51abb3c bitbake: toaster: remove custom images from Image Recipes
Fix the view of 'Image Recipes' under 'Configuration' to only show
image recipes that are not customised since custom images have their
own page.

[YOCTO #9111]

(Bitbake rev: 18a93b360301a5497d5c8ef74ab71f374f2ad210)

Signed-off-by: Dave Lerner <dave.lerner@windriver.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:29 +00:00
Elliot Smith 98d462c8e6 bitbake: toaster: show suffix for image files and basename for artifact files
The build dashboard doesn't show image and artifact files correctly,
as it shows the full filename for images and the filename plus
path relative to DEPLOY_DIR for artifacts.

Instead, show just the suffix for image files, and the basename
for artifact files.

(Bitbake rev: 8084dcdc283b4dc170f066c202f89d56ce1abbef)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:28 +00:00
Belen Barros Pena 88b56603ce bitbake: toaster: add missing link to image recipe details
In the 'New custom image' page, each image recipe name listed should
link to the corresponding image recipe details page, so that users can
look into what packages are installed by a certain image, and decide
based on that if they want to customise it or not.

This patch adds that missing link.

(Bitbake rev: a481af693bfef0171732c18c298e285986b82de3)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:26 +00:00
Belen Barros Pena 25b179d5c3 bitbake: toaster: adjust the search field width
The search field at the top of our tables was using one of the Bootstrap
classes for text field sizing. Those classes are a bit rigid, resulting
in text fields too wide that made other table controls wrap.

Setting a maximum width to the search form using one of the span classes,
combined with a % width css declaration, make for text fields that
adapt a bit better to the horizontal space available in each table.

(Bitbake rev: 7833fab2e03f2d9a01ab9ad0a13c190382098b5e)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:24 +00:00
Belen Barros Pena a97081ba55 bitbake: toaster: make 'configuration' the first tab
Our project pages have 4 tabs: builds, configuration, import layer and
new custom image. Even though we treat the 'configuration' as the
default tab, it comes second after the builds tab.

That's a bit strange: the default tab should be the first one listed.
This patch changes the tab order to put 'configuration' first.

(Bitbake rev: ccb90019489c2c324c2a5a60295e02280a2ec18f)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:23 +00:00
Belen Barros Pena e1fc3190b7 bitbake: toaster: link to configuration in all breadcrumbs
The existing breadcrumb does not always provide a link to the project
configuration page. When you are in the build history pages, you must go
back to the builds information first, and from there access the project
configuration. That feels very long.

Change the breadcrumb so that the project name item always provides a
link to the project configuration.

(Bitbake rev: 9910f3f292d35fc91215d550c5f123dcf18ab35d)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:21 +00:00
Belen Barros Pena df2808f5f6 bitbake: toaster: reduce max height of modal dialogs
Now that we use modal dialogs to display dependency information for
packages, we are hitting their maximum height relatively often. It is set
by default to 400px, which makes it a bit tight at a 1280x800 viewport
size.

Reduce the maximum height to 300px to make things a bit more
comfortable.

(Bitbake rev: e36001d61768979d66cba0f3d4f5a2aaf4af2cb7)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:18 +00:00
Belen Barros Pena 6c51f0850f bitbake: toaster: disable add layer button on click
The 'add layer' button in the project configuration page remains enabled
after you add a layer. If you click it again, the same layer you just
added is added again.

This patch disables the 'add layer' button on click, to avoid this bit
of weirdness.

[YOCTO #8905]

(Bitbake rev: 63705f60035884a810fdd36e5a3fe10e411f23c7)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:16 +00:00
Belen Barros Pena d4a663a701 bitbake: toaster: apply error class to name field
The form for naming new custom images shows you an error message when
the name already exists or you include an invalid character in it. But
when an error appears, the input field was missing the red highlight.

This patch applies the right class to the form controls whenever an
error message is shown.

(Bitbake rev: df342e7662179410467c47cd870180ea75f863d4)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:15 +00:00
Belen Barros Pena 48f0ae2c12 bitbake: toaster: fix custom image name form
The placeholder text in the form where you name your new custom images
didn't display fully. This patch fixes the styles so that the text shows
properly. It also changes the text itself to make it a bit shorter.

[YOCTO #9122]

(Bitbake rev: 9df802182b0b96295b148a1681c2265e72d8306b)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:12 +00:00
Belen Barros Pena 07eb4f274c bitbake: toaster: comment out project release change
Although the support for building more than one release and how we
handle the build directories is the subject of lively discussion, we all
seem to agree on removing the ability to change the release of a
project. The feature is currently not working but exposed to users,
which is not a happy state of affairs.

This patch comments out the controls that give access to the release
changing functionality to hide them from users, but does not touch
anything else. Once all moving pieces start to settle down, we can make
a final decision regarding this feature, and clean up the code
accordingly.

[YOCTO #8917]

(Bitbake rev: 3a8c6f7155517cd61a160595b81e7bed84ba4eaf)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:11 +00:00
Richard Purdie 12ade9b093 bitbake: fetch2/npm: Add mirroring support for npm fetcher
(Bitbake rev: bff46c614d3f9cc18a1c8908c47842712e0e3a8c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:09 +00:00
Brendan Le Foll ca5b6d66e0 bitbake: fetch2/npm: Add npm fetcher
npm fetcher with support for shrinkwrap files and lockdown files to easily
download and install an npm package with strict dependency resolution.

The SRC_URI should be in the format of:
SRC_URI = "npm://registry.npmjs.org/;name=${PN};version=${PV}"

To add a shrinkwrap and lockdown file use:
NPM_SHRINKWRAP := "${THISDIR}/${PN}/npm-shrinkwrap.json"
NPM_LOCKDOWN := "${THISDIR}/${PN}/lockdown.json"

(Bitbake rev: dec75bbc5d075acb322dad8b1c40d6bd518dc9fd)

Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:07 +00:00
Brendan Le Foll 813bd1f806 bitbake: utils.py: Add sha1_file call
This is useful as npm-lockdown uses sha1 because npm releases the sha1 of
packages and whilst this is undocumented it seems no other algorithm is
supported

(Bitbake rev: fd5d9011f6dd7029895b64d8a02d33185b9aa8ae)

Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:05 +00:00
Michael Wood 8c0ba8d9af bitbake: toaster: toastergui Fix invalid char test and implementation
This test was passing but because the assertion was the wrong way round
and should have expected the first one to pass and second one to fail,
in reality both were failing as the method for checking the invalid char
was incorrect.

(Bitbake rev: 932a92b8130d4815656dc885f0c6e4afa4502022)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-19 15:38:49 +00:00
Michael Wood 913e9b1cbc bitbake: toaster: PackagesTable show only installed packages
When showing the package list for the image recipe details only show the
packages which are installed in the image rather than all the packages
which are produced.

[YOCTO #9108]

(Bitbake rev: be7dca6c7607c0d13151c2d3f7ad7adcdf365076)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-19 15:38:48 +00:00
Michael Wood 94bca58067 bitbake: toaster: toastergui unit tests convert to use fixtures
Instead of having to manually write up in the setUp function all the
different objects to create in the database, just specifiy them in a
fixture and load it. This has the advantage that it can be used on a
live server by using the ./manage.py loaddata command and can then
be inspected.

(Bitbake rev: 8a653ad8b075892d24d15696b3070c5a28a4fbe2)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-19 15:38:48 +00:00
Michael Wood 8796ac8a55 bitbake: toaster: SoftwareRecipesTable apply default order_by
Make sure the default orderby for the SoftwareRecipesTable is applied

(Bitbake rev: 1688608b537d8de840c6d1e4802ae41ca872e5bf)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-19 15:38:48 +00:00
Michael Wood 8469e58025 bitbake: toaster: orm migrations Sort out migrations mess
We messed up the migrations by squashing some of the image
customisation model definitions into the initial migration which
has meant some irreversible operations on mysql took place.
This deletes, re-orders and fixes the migrations.

If your schema is up to date you may want to use ./manage migrate
with --fake or --fake-initial to avoid re-applying migrations.

[YOCTO #9116]

(Bitbake rev: 19bd63fc3a28dcbd0f531a5b06a037da34568bac)

Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-19 15:38:48 +00:00
Richard Purdie 4e5e501155 bitbake: data_smart: Drop default expand=False to getVarFlag [API change]
At some point in the future, getVarFlag should expand by default. To
get there from the current position, we need a period of time where the
expand parameter is mandatory.

This patch starts that process. Clear errors will result from any code
which doesn't provide this. Layers can be fixed with an expression
like:

sed -e 's:\(\.getVarFlag([^,()]*, [^,()]*\)):\1, False):g' -i `grep -ril getVar *`

(Bitbake rev: aa3faebdf6af66ab34f74d328b2113de0b08c7ee)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:48:00 +00:00
Richard Purdie c7610aa353 bitbake: data_smart: Drop default expand=False to getVar [API change]
At some point in the future, getVar should expand by default. To get
there from the current position, we need a period of time where the
expand parameter is mandatory.

This patch starts that process. Clear errors will result from any code
which doesn't provide this. Layers can be fixed with an expression
like:

sed -e 's:\(\.getVar([^,()]*\)):\1, False):g' -i `grep -ril getVar *`

(Bitbake rev: fab717d303df0bcef737661f6917f275f35215a4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:48:00 +00:00
Markus Lehtonen 4f0ab27c6a bitbake: SignatureGeneratorBasic: make checksum cache file configurable
Define a new bitbake configuration variable BB_HASH_CHECKSUM_CACHE_FILE
that can be used to define the cache file to use for file checksum
cache.

(Bitbake rev: a965b390d6240e279c190b92b17c0573e9bd604c)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:41:16 +00:00
Markus Lehtonen 0cdf1934fd bitbake: MultiProcessCache: make cache filename configurable
If no cache file name is given a default from class variable is used,
like before.

(Bitbake rev: 2602a312818f564961de7dfa63c429d45ff9e5ac)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:41:16 +00:00
Markus Lehtonen ca552bb4b1 bitbake: FileChecksumCache: add get_checksums() method
Move the local file checksum functionality from bb.fetch2 into
bb.checksum module.

(Bitbake rev: 4f60933283f377d68f191db849dac6c1dc7a0aed)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:41:16 +00:00
Markus Lehtonen 8f61f2d881 bitbake: bb/runqueue: save task file dependency cache onto disk
Before this patch the usage of cache was quite useless as the file
checksums were not actually cached on disk but re-calculated every time.
This patch utilises the new writeout_file_checksum_cache() method of the
SignatureGenerator class to do the job.

(Bitbake rev: 5ac9cbf405841ed3f65e6f99a3cee032567fb182)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:41:16 +00:00
Markus Lehtonen 5177b1e4ac bitbake: SignatureGenerator: add method for saving the file checksum cache
Extend the API in order to be able to write out the file checksum cache
onto disk. SignatureGeneratorBasic class now implements a method that
update the fetcher local files checksum cache with the task file
dependency checksums.

(Bitbake rev: ecdabd321d48fa367b89ebffc00aa525b6eaa95c)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:41:15 +00:00
Markus Lehtonen 97617fd675 bitbake: bb/cache: drop some unused arguments
Drop unused 'd' argument from the cache save methods, simplifying the
API.

(Bitbake rev: 81bc1f20662c39ee8db1da45b1e8c7eb64abacf3)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:41:15 +00:00
Mirela Rabulea 5a87d8c477 bitbake: Allow Hob to run images on a custom simulator, other than qemu
The current behavior of Hob is that there is a "Run Image" button which becomes visible only for qemu images.

My suggested change is:
- if an image is selected and it is qemu-compatible, let the "Run image" button be named "Run qemu image"
- if an image is selected and it is not qemu-compatible, let the same button show up with the name "Run custom image", and besides that, an option shows-up to allow the selection of the custom script (by default it points out to runqemu script) to be used for launching this custom image

Note: in case there is more than one toggled image (qemu runnable or deployable), when the user clicks the "Run custom image" button, a dialog will be presented, allowing to choose between any of the existing images.

[YOCTO #8940]

(Bitbake rev: cc4cfc2370297b8feb2dc39d4262e73adf06c09a)

Signed-off-by: Mirela Rabulea <mirela.rabulea@nxp.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:41:15 +00:00
Christopher Larson a25858982a bitbake: bb.ui.knotty: prefix task messages with recipe/task
An example prefix: `perl-5.22.1-r0 do_compile:`

(Bitbake rev: 792b759e59e31d2e43d525a6e50d866b4f51f072)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-16 09:04:23 +00:00
Christopher Larson 4bf8b21ab8 bitbake: Move bb.{debug,note,..} into their own logging domain
This lets us filter and use -l to show messages from that source specifically.

(Bitbake rev: 7946927156dec33364418988eb921ddb273660eb)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-16 09:04:23 +00:00
Richard Purdie 8b217208cf bitbake: knotty: Set exit failure code on runQueueTaskFailed events
If the worker segfaults, we may never see a TaskFailed event from it, only
a runQueueTaskFailed event. In this case, return_value isn't getting set
leading to an incorrect exit code from bitbake. Fix by setting return_value
in both places.

(Bitbake rev: e5dd50e0d95d532fe31dde61f8c6b1a7a72321e9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-15 16:48:12 +00:00
Richard Purdie a9223e20d1 bitbake: taskdata: Fix traceback issue with missing provider
If there is a missing provider and we're using "-k" mode alongside "-w",
we could get a traceback since there was no provider. Add tests to avoid this.

(Bitbake rev: 90a4805e4e770a433b4394ea99792731e9a4b546)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-15 16:48:12 +00:00
Richard Purdie 7593756cbb bitbake: cooker: Improve cache handling
We're seeing issues where the self test, which uses tinfoil doesn't
notice the changed contents of include files. The issue is
cached_statements in the parser being reused when the files have changed.

Whilst looking at this, I realised there were some other issues:

* We need to also invalidate the mtime cache when cooker restarts
* We should pass full filenames to the file invalidation code
* We should process cached_statements as part of inotify invalidation

With these fixes, the caching is more reliable for memory resident
bitbake too. It does raise some questions about cache validation and
lifecycles and indicates bitbake does need more work in the area,
preferably with the removal of the globals. This at least highlights
and works around some of the current issues.

(Bitbake rev: 3f507ff8bc467fba936cf3f31bb8ea8e02f168e8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-15 16:48:12 +00:00
Richard Purdie 1c2d632a02 bitbake: Bump version post release to 1.29.0
(Bitbake rev: da1107a2c2331b3335c7bd46821edcb1933a0ff8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-12 13:44:22 +00:00
Ed Bartosh f9f93ae856 bitbake: cooker: gracefully shutdown parsers
CookerParser.shutdown code doesn't do all required work to shutdown
parser processes. As a result bitbake hangs if interrupted during
parsing. Putting None into the parser_quit queue should fix this issue
as it makes parsers to quit main loop.

(Bitbake rev: f67307977e8f089ce6d208d3e9de2a6a1768757e)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11 22:34:19 +00:00
Elliot Smith 1f7f077466 bitbake: buildinfohelper: unset brbe variable when build finishes
The brbe variable is set on the bitbake server when the Toaster
UI starts. This enables Toaster to associate events with the
correct build and build environment.

However, the brbe variable is also used when a build starts to
identify whether a new build needs to be created, or an existing
one looked up. This causes a bug for command-line builds which
happen after a Toaster-triggered build: because the brbe variable
is never unset on the server or the buildinfohelper, the new
command-line build events are treated as originating from the
previous build.

Ensure the brbe variable is reset when the buildinfohelper "closes"
a build, so that each build then either sets the brbe variable
(Toaster-triggered builds) or leaves it blank (command-line builds).

Also modify the localhostbecontroller so that the brbe variable
is not set on the server and not looked up from the server. This
ensures that it is only set when the triggerBuild() method is
called, and that it remains as None for command-line builds.

[YOCTO #9021]

(Bitbake rev: 4a6a8d0074f62208d843b06344be31ae73d9b745)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11 22:34:19 +00:00
Mihail Stanciu 19045bab71 toaster: tests Remove symlinks from toasteruitest folder
Remove symlinks in the UI tests folder as they are causing
problems for bitbake upstream.

[YOCTO #8787]

Signed-off-by: Mihail Stanciu <stanciux.mihail@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11 17:07:39 +00:00
Richard Purdie 400f53003e bitbake: build: Improve python execution tracebacks
If something fails in a exec_func_python() the current stack trace shows
incorrect filenames and linenumbers. For example:

The stack trace of python calls that resulted in this exception/failure was:
File: '/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb', lineno: 200, function: <module>
     0196:                chksum = bb.utils.sha256_file(fn)
     0197:                f.write('%s\t%s\n' % (chksum, os.path.relpath(fn, baseoutpath)))
     0198:
     0199:
 *** 0200:copy_buildsystem(d)
     0201:
File: '/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb', lineno: 9, function: copy_buildsystem
     0005:IMAGE_FEATURES += "splash package-management x11-base x11-sato ssh-server-dropbear hwcodecs"
     0006:
     0007:LICENSE = "MIT"
     0008:
 *** 0009:inherit core-image
     0010:
     0011:IMAGE_INSTALL += "packagegroup-core-x11-sato-games"
File: '/usr/lib/python2.7/subprocess.py', lineno: 535, function: check_call
     0531:    The arguments are the same as for the Popen constructor.  Example:
     0532:
     0533:    check_call(["ls", "-l"])
     0534:    """
 *** 0535:    retcode = call(*popenargs, **kwargs)
     0536:    if retcode:
     0537:        cmd = kwargs.get("args")
     0538:        if cmd is None:
     0539:            cmd = popenargs[0]

The problem is the use of "FILE" to obtain the current filename. Instead,
we therefore inject the function being executed into the methodpool which
allows us to correct its linenumber and filename information. We can then
clearly mark the initial piece as autogenerated and the rest of the linenumber
and filename information should be correct. Afterwards the trace starts:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:copy_buildsystem(d)
     0003:
File: '/media/build1/poky/meta/classes/populate_sdk_ext.bbclass', lineno: 66, function: copy_buildsystem
     0062:    import glob
     0063:    import oe.copy_buildsystem
     0064:    import subprocess
     0065:
 *** 0066:    subprocess.check_call("foo")
     0067:
     0068:    oe_init_env_script = d.getVar('OE_INIT_ENV_SCRIPT', True)
     0069:
     0070:    conf_bbpath = ''
File: '/usr/lib/python2.7/subprocess.py', lineno: 535, function: check_call
     0531:    The arguments are the same as for the Popen constructor.  Example:
     0532:
     0533:    check_call(["ls", "-l"])
     0534:    """
 *** 0535:    retcode = call(*popenargs, **kwargs)
     0536:    if retcode:
     0537:        cmd = kwargs.get("args")
     0538:        if cmd is None:
     0539:            cmd = popenargs[0]

We can't inject into methodpool at parsing time, since there may be
_append or other override operations against the function before its
execution.

(Bitbake rev: fae153095d23157dd7e72c29f683f86149ee33a8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 15:47:11 +00:00
Richard Purdie aece748762 bitbake: build/data: Don't expand python functions before execution [API change]
Right now, if you have some python code like:

X = "a"

def somefunction(d):
    d.setVar("X", "b")
    d.setVar("Y", "${X}")

then any sane person would expect that Y = "b" at the end of the
function. This is not the case, Y = "a".

This is due to the python function being expanded before execution, the
executed code would read d.setVar("Y", "a"). This understandably
confuses people, it also makes it near impossible to write ${} in a
python function without unintended things happening.

I think there is general agreement we should fix this and standardise
on non-expansion of python functions. We already don't expand anonymous
python (mostly).

I've checked OE-Core with buildhistory before and after this change and
there were a small number of issues this exposed which I've sent
patches for.

I propose we default to not expanding python code and then deal with
any consequences from that if/as/where identified. This will improve
new user understanding and usability of the system, it also allows
several long standing weird expansion issues to be fixed.

(Bitbake rev: 8bf33a8e92c0e188fa392030025756196c96fcbb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 15:47:11 +00:00
Richard Purdie e39cfb1f9c bitbake: cooker: Don't expand python functions in variable dumps
We don't want to expand python functions since they aren't expanded
at execution time (e.g. anonymous python). They can also have side
effects.

This function is primarily used by toaster for variable dumps for later
display. The lack of expansion of python functions won't matter in this case
and actively helps some variable handling (e.g. SRCPV).

(Bitbake rev: 3f5520b4844a4bdd615046479ba08ed192bdc8cd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 15:47:11 +00:00
Richard Purdie f652b6b9d0 bitbake: data: Don't expand python functions for variable dependencies
Expanding python functions for variable dependencies doesn't really make sense,
not least since this causes execution of any inline python, it also makes it
impossible to write expressions like d.expand("${X}") of d.setVar("X", "${Y}")
which may have the wrong values if expanded now.

This starts to standardise the approach across bitbake for handling python code.

(Bitbake rev: 765a2480dbe288f64562a9611dd93b6b6dd0a64e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 15:47:11 +00:00
Richard Purdie d3e0c448ea bitbake: data_smart: Avoid expanding anonymous python functions
We don't expand anonymous python before execution, so nor should
we do this when calculating checksums for them.

(Bitbake rev: 5f10987edda35b08970a6dd6ccf9febad271ce3e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 15:47:11 +00:00
Michael Wood e0eb2ea488 bitbake: toaster: models Remove manual transaction control from lsupdates
Revert "toaster: models.py do not use transactions".
This reverts commit e248a503ba
(Bitbake rev: 48d0d510816346073f135bb86fb7904fdb50bb27)

Manually managing the database transactions caused more
problems than it temporarily solved so we return control back to
Django's orm.

[YOCTO #8796]

(Bitbake rev: 25c531915b6f8f79a0822996ceb97f90483e359f)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:33:40 +00:00
Belen Barros Pena 48622e1d6b bitbake: toaster: build section Improve display of builds when > 1 targets
* Display always the first target in alphabetical order to match what we
do in the breadcrumbs and the build dashboard heading

* Remove the extra space between the '+' and the additional number of
targets

* Make sure the tooltip with the full target list takes the Bootstrap
tooltip styles

* Replace the word 'targets' in the tooltip with 'recipes', since that's
how we call build targets everywhere else in Toaster

(Bitbake rev: 3b8747d0af4b9164e973940ed97751c951e74110)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:33:40 +00:00
Belen Barros Pena 4d0ba0fca8 bitbake: toaster: templates make build data breadcrumb consistent
The pages in the build data section of Toaster showed different
breadcrumbs: in some pages the machine was displayed, but not in others.
For builds with more than one target, some pages showed the first
alphabetical target (the correct behaviour), others didn't.

This patch removes the inconsistencies, showing exactly the same
breacrumb across all pages in the section.

The patch also removes the extra space between the '+' and the number of
targets when the builds have more than one target.

Remove an unneeded debug message

(Bitbake rev: 9cdbb543311b6f4a8a88c27fc157d998242444ee)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:33:40 +00:00
Richard Purdie 99184d7879 bitbake: BBHandler/ast: Merge handMethod and handleMethodFlags
The functionality overlap between these two functions is significant and
its clearer to handle both things together since they are intimately
linked. There should be no behaviour change, just clearer code.

(Bitbake rev: 391aa4afc91be90d8d3ee47e1bf797d6ebe61a71)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:32:46 +00:00
Richard Purdie 6ba69b4383 bitbake: utils: Drop datastore function inspection during exception
When we use functions from the data store, they now have correct line number
and filename information. This function would attempt to correct line numbers
which doesn't need correcting, leading to misleading messages to the user.
Therefore remove this code as being obsoleted.

(Bitbake rev: 918bec86bc8ee94feb82380ff410d9fdcbe9e720)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:32:46 +00:00
Schroeder, Henning f8a44b160d bitbake: cooker: extended dot styling
Extended the dot styling of dependencies created by bitbake -g in dot syntax to differentiate between the various kinds.
  depends: solid
  rdepends: dashed
  rrecommends: dotted

The change observed is that depends get an explicit style which is the same as dot default behavior and the runtime recommends get
dotted while before they were dashed. This helps to distinguish them graphically as well as eases post processing by script.

(Bitbake rev: 86e78e0ca7aa5452411f35239942ecee3d8824ec)

Signed-off-by: Henning Schroeder <henning.schroeder@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:32:46 +00:00
Michael Wood 30c132bc47 bitbake: toaster: Enable Image Customisation feature
Remove environment variable to toggle Image customisation feature

(Bitbake rev: 2e9c86229b8f924a5b62987f4b166f63392f12e8)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:22 +00:00
Michael Wood 5e14a8f2fe bitbake: toaster: xhr_customrecipe_packages Add dependencies to included packages
Add the dependencies of packages which are added to the
CustomImageRecipe. Currently just handle the first tier of dependencies
as this is what we show in the UI.

(Bitbake rev: 5c44609a9bf9fb23241b7dd7c58b08901d75008d)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:22 +00:00
Michael Wood 749f5a6f1f bitbake: toaster: orm generate_recipe_content only exclude locale packages
Allow package groups in our custom image recipe. Excluding them creates
more undefined behaviour than including them at this stage. Also update
to use convenience method for returning all packages.

(Bitbake rev: 8c2e8a13badacb816c4b1178b6661600008b38af)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:22 +00:00
Michael Wood 626941104f bitbake: toaster: customrecipe page Add last successful build link and conditionals
Add link to the last successful build if there is one and add
conditionals for the ancillary recipe metadata.

(Bitbake rev: 4660aaf1c6775270f8f3d0afbb7fa2ee7a2a1563)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:22 +00:00
Michael Wood 8d5b61e495 bitbake: toaster: models Add update_package_list for CustomImageRecipe
Add a method to update the packages included list from the last
build, this effectively "synchronises" the package list from what
we think will happen at the Customise image stage with what
actually was produced with a build. It's not ideal to have this
function here but we also need to make sure that no race condition
of the user accessing this list and it being updated occurs.

(Bitbake rev: 8cf6e67a955574b33856a082bdadf3194f2b6ba4)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:22 +00:00
Michael Wood 86db0bd7f2 bitbake: toaster: orm Add last_updated field to CustomImageRecipe
Field to keep track of when the package list for the CustomImageRecipe
was last updated from a build.

(Bitbake rev: 4bd4e49f13a7625997a43f3b2e67ed42c3c8e08b)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:21 +00:00
Michael Wood 18d8b17052 bitbake: toaster: models add get_last_successful_built_target method
Add a convenience method to get the last successful build
target for a CustomImageRecipe.

(Bitbake rev: 4dde3d830cd38bbe306d83629dcb80da5fc9b027)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:21 +00:00
Michael Wood 8885b7bfd3 bitbake: toaster: pkg_dependencies_popover just show direct dependencies
In the dependencies popover just show direct dependency in the list
rather than recommends, conflicts etc

(Bitbake rev: e69b00532b011327bc2495a6fb52cfe98f0f897d)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:21 +00:00
Michael Wood 40f6effec4 bitbake: toaster: models add all_depends method for Package_DependencyManager
This convenience method returns just the Package_Dependency for
the package which are regular dependencies i.e. not RECOMMENDS or
any other types.
(Bitbake rev: bd76c22fe2aa06690b4ee25de69219ac0bf6b4d6)

Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:21 +00:00
Michael Wood a8ab1c6b25 bitbake: toaster: buildinfohelper CustomImagePackage update dependency info
Instead of keeping the original dependency information for the pool of
CustomImagePackage reset it with each new build.

(Bitbake rev: a0b97ffc7a468bad081ce3276c74728bf6830250)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:21 +00:00
Michael Wood 0fee829932 bitbake: toaster: newcustomimage_modal add frontend name validation
Add front end handling of validation response from create new
CustomImageRecipe api.

(Bitbake rev: eff66b502df8e001cd0abc25bcbd742687169619)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:21 +00:00
Michael Wood cb6d290d0b bitbake: toaster: API CustomImageRecipe check the recipe name supplied is valid
Check that the name for a new CustomImageRecipe doesn't already exist in
the project or in the database of existing recipes (e.g. from the layer
index). Also restrict the characters entered for the recipe naming
convention.

(Bitbake rev: f290d428460a07e73050ff613bc222cc8c04f5ec)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:21 +00:00
Michael Wood 5634a251ce bitbake: toaster: views CustomRecipe API add size information to the package lists
Add the file size of the packages and the total to the JSON response.

(Bitbake rev: bbbd304c49b0940a695d15273934edff95d70836)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:21 +00:00
Michael Wood 6fbceb0f12 bitbake: toaster: models Invalidate ToasterTables cache when a m2m field changes
Whem a m2m field changes we need to clear the ToasterTables cache as
this can affect the state of items in ToasterTables.
For example the CustomImagePackages being added or removed from a custom
image recipe.

(Bitbake rev: c9d7b68ee0186a71e8e75a5d87122a0328001515)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:21 +00:00
Michael Wood 998f9af193 bitbake: toaster: customrecipe Add dependency tracking to package selection
Update the states of the packages in the package selection UI to reflect
whether it's likely that 1st level dependencies for the package will be
also added.

(Bitbake rev: 119569d83c3fb1d1bd162624819b3f9c63a791c4)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:21 +00:00
Michael Wood 9976e4f169 bitbake: toaster: tables move template logic into the pkg_add_rm_btn
Instead of defining this as a string it's sufficiently large enough to
warrant its own file.

(Bitbake rev: 6b39423fe5a3ed30289a8b303329a5725f7d273b)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:20 +00:00
Michael Wood d77c247230 bitbake: toaster: CustomImageRecipe generate overwrite IMAGE_FEATURES
If we're breaking up an image recipe's packages we will also need to
override IMAGE_FEATURES to make sure the customisation is not altered

(Bitbake rev: 9fd7b05dc0cf9240f7c8e3dc77b009064fd2b0cb)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:20 +00:00
Elliot Smith 481dc11547 bitbake: toaster: make locale packages uneditable in custom image page
When the packages associated with a custom image recipe are
shown in the customrecipe editing page, locale packages are
shown in the same way as all other packages. This gives the false
impression that these packages can be removed, when in fact
they are automatically added due to the IMAGE_LINGUAS build
variable.

Modify the customrecipe page so that locale packages cannot be
removed, and provide some help text explaining why.

[YOCTO #8927]

(Bitbake rev: b2208e53c00a67a7d0345e7378e6806b8ae40fb4)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:20 +00:00
Elliot Smith a757d397db bitbake: toaster: include locale and packagegroup packages in custom image
The custom image editing page doesn't show locale and packagegroup
packages: they are filtered out of the queryset used to populate
the ToasterTable.

Rather than filtering these packages out, include them in the list
of packages which are shown.

(Bitbake rev: 38a753e7b2e9ede326856b830b25e13bdd6d0d9b)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:20 +00:00
Elliot Smith baac4589ae bitbake: toaster: update custom image package table filters
The custom image package selection filters were using the old
ToasterTable filter approach, which caused the table filter to fail.

Amend the table to use the new ToasterTable filtering API to fix
this.

(Bitbake rev: 72a4cb30842fd053e46dc56df222729cbe735162)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:20 +00:00
Elliot Smith efbffe3c29 bitbake: toaster: move recent builds query to model
The progress updater for the recent builds section makes a JSON
call to the project view URL to get progress for each build.
However, conversion of the builds pages to ToasterTable broke this,
as the JSON response no longer contained the data necessary to
populate the progress bars.

Move the recent builds query to the Build model, so that it is
accessible to the ToasterTables using it ("project builds" and
"all builds"), as well as to the "project" view.

Modify the code in the recent builds template to use the slightly
different objects returned by the recent builds query on Build.

(Bitbake rev: 5189252635ddc7b90c9a43aaed9f196c31e1dcad)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:20 +00:00
Elliot Smith b51478582f bitbake: toaster: update customimagerecipe migration
When applying migrations, Django shows this warning:

"Your models have changes that are not yet reflected in a migration,
and so won't be applied."

This is because the customimagerecipe model has changed, but those
changes are not covered by a migration.

Add the missing migration to clear this warning.

(Bitbake rev: df8185fcbd84061976d91b03b2a9268b319a6184)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:20 +00:00
Elliot Smith df58f5b70b bitbake: toaster: add merge migration to resolve conflict
Django detects a conflict between a migration added to support
image customisation and another migration which supports PROVIDES.

Add a merge migration to resolve the conflict (as suggested by
Django).

(Bitbake rev: a26bfd9d2490dc0fd90bf6d1690e63ac26001559)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:20 +00:00
Michael Wood 38f4913270 bitbake: toaster: orm generate_recipe_file_contents Handler for require recipe
Add a special case for when the recipe we have based a custom image
recipe on requires another recipe.
In this case we need to adjust the file location to be able to
require
the recipe when we're in the toaster-custom-images layer.

For example:
"require core-image-minimal.bb" is changed to:
"require recipes-core/images/core-image-minimal.bb"

(Bitbake rev: 26025e1ea49b3ebfcfd508d1608fa8c9e722ad03)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:20 +00:00
Michael Wood 769017e477 bitbake: toaster: project builds Poll the server to get latest progress for build
Poll the server for the project build progress value. This is something
that will need to be re-done once we have a proper API for this on the
server side.

[YOCTO 8328]

(Bitbake rev: ec467e43c39eadf02412b89db10c09ed78a5a9f5)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:20 +00:00
Michael Wood 971d65c614 bitbake: toaster: localhostbectrl Update the dirpath of customrecipe's base layer
We need to know the location of the based_on recipe's layer on the
file system before we try and generate the custom image recipe. As
we read the recipe to make the custom version.

(Bitbake rev: e6a7cacbddd1df5bac0b79384199cf7264c5bbd5)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:19 +00:00
Michael Wood 6d9f342e12 bitbake: toaster: tables Check layer presence in project for customise_btn
Make sure we send the current list of layers to the customise button to
be able to know whether it should be set as an add layer button or a
customise button on the New custom image page.

(Bitbake rev: 5ddb35c98b609d85f97d482b54cabe3a2812afe6)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:19 +00:00
Michael Wood 76c0008014 bitbake: toaster: toastergui tests Add addtional data to the setUp for new tables
Add additional data to the setUp to be able to test all the tables for
Image Customisation. Also add the name of the table being tested to the
num of rows assertion.

(Bitbake rev: dfcbcf789cf3f0733ca26b0601fdf97ce4291674)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:19 +00:00
Michael Wood 70a078ee85 bitbake: toaster: tables SelectPackagesTable rename recipe_id to custrecipeid
Rename the recipe_id to custrecipeid to avoid confusion about which type
of object we're going to be accessing. This means that in the unit tests
for tables we can pass a different kwargs for custom recipes vs normal
recipes.

(Bitbake rev: ae3301a1047b3efb4b340b50a10d5d585b7333da)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:19 +00:00
Michael Wood 7e4c231ca2 bitbake: toaster: toastergui tests Update package test to use CustomImagePackage
Update test for adding and removing a package from a CustomImageRecipe
so that it uses the CustomImagePackage and correct fields for the packages
included. Change the test for error condition to use an invalid package
id as ManyToMany remove() on package that isn't in the relationship does
not throw an error.

(Bitbake rev: daccb2978f833a9e7af270160331da3e9a158219)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:19 +00:00
Michael Wood 4b3c9d61dc bitbake: toaster: customrecipe Add further front end features using new API
This adds some basic package dependency hint modals when you add and
remove a package. It also makes sure that if the CustomImageRecipe has
no current included packages that we go and check this with the server
to see if a relevant build has taken place which will provide this
information.

[YOCTO #8082]

(Bitbake rev: 418f5509e74d46d36a8eb966a245083006e5f4ba)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:19 +00:00
Michael Wood b213907afe bitbake: toaster: xhr_customrecipe_packages add GET info for package response
Add response for GET to the xhr_customrecipe_packages ReST API

/xhr_customrecipe/<recipe_id>/packages/<package_id>
Thie response includes the id, name, version and dependency information
for the package.

(Bitbake rev: c45791fc85d26c43b0a3b0a459111d2ff5583540)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:19 +00:00
Michael Wood a9668eeb74 bitbake: toaster: xhr_customrecipe_id change to use CustomImagePackage
Instead of doing a shallow copy of the package into the
CustomImageRecipe when we add packages we can now use the
CustomImagePackage as a M2M field on the Package to CustomImageRecipe.
Also switch to using Target_Installed_Package as the method to retrieve
the package list from the build.

(Bitbake rev: 4ebc81823b3aec6ecf38835acad5263a81eb41c5)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:19 +00:00
Michael Wood 439314c1b2 bitbake: toaster: API allow CustomImageRecipe to be updated after creation
When we create a CustomImageRecipe we create a Layer_Version and
Recipe for that Recipe to be in, we only need one Layer_Version for our
Recipes so if that Layer_Version is updated by building it we need
a slightly more custom version of get_or_create to take into account the
fields which we expect can change but still mean that the object we want is
valid and doesn't need to be created.

In the Recipe case this is when we're updating an existing
CustomImageRecipe as we allow people to create a recipe even when the
based on recipe hasn't been built so we need to update it once a build
has happened.

(Bitbake rev: 0fe2c72ab82c6de2825a390fbb460b892a7a9cfc)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:19 +00:00
Michael Wood 9ea4de6d80 bitbake: toaster: tables Change SelectPackagesTable to use ProjectPackage
This changes the SelectPackagesTable to use the ProjectPackage table
instead of very large expensive queries to retrieve a list of currently
available packages for the project.

(Bitbake rev: 4b4b7e28d602ac5283659f806d695cc0451d292e)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:18 +00:00
Michael Wood 20f400b7bd bitbake: toaster: tables add recipe download link to CustomImagesTable
Add the download recipe link and fix the package count field.

(Bitbake rev: 85891e5320014f363dba093ac2db681d55375ee3)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:18 +00:00
Michael Wood 1c9ce1ca73 bitbake: toaster: newcustomimage_modal use libtoaster method for new CustomRecipe
Use libtoaster.createCustomRecipe rather than own implementation of this
function.

(Bitbake rev: 74fa98752b1cf1ad18d35ab6dd25fe7e409133c5)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:18 +00:00
Michael Wood 8b1d0438bc bitbake: toaster: libtoaster Add createCustomRecipe method
This adds the function to call the ReSt API to create a custom image
recipe.

(Bitbake rev: 03e7949f538733f682a05d0c318cf2f4cd64cbf5)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:18 +00:00
Michael Wood 32048fa71f bitbake: toaster: orm Add convenience method to get all pkgs in a CustomImageRecipe
Returns a queryset of the all the packages that we expect to have in a
CustomImageRecipe.

(Bitbake rev: 8b03bbae12ec077151c97579e329d89667040a78)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:18 +00:00
Michael Wood c80b7dfee2 bitbake: toaster: orm get_project_layer_versions to return layer_version objects
Instead of returning layercommits return the actual Layer_Version objects
for the layercommit as these are the useful objects which contain the
metadata.

(Bitbake rev: 480f0d307a3183df16fd383c7a3f96f00d09fbd4)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:18 +00:00
Michael Wood 796e3481ad bitbake: toaster: toastergui tests Add unit test for download custom recipe
Add unit test to test downloading end point and the basic content of the
custom recipe that is generated.

(Bitbake rev: 1e78d1ca2f5b02d80ab037c6f5f81d18195fc817)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:18 +00:00
Michael Wood 04d8c94564 bitbake: toaster: toastergui tests Update to reflect changes to CustomImageRecipe
Now that CustomImageRecipe inherits from Recipe make sure that the
accessors and the required values for Recipe are now setup correctly.

(Bitbake rev: e958921e15a3c3e5a6b7c27ebe37fdf1f551f198)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:18 +00:00
Michael Wood 4e8a0aa66e bitbake: toaster: views xhr_customrecipe_packages clean up API
- Fix generic variable names such as "object" and "values" when not
  needed.
- Use try catch instead of a queryset filter to return the custom recipe
  object
- Be explicit about the fields returned for the custom recipe info field
- Remove redundant new_package field

(Bitbake rev: a1a69903a94264377666730b1eb4599e6f3b4398)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:18 +00:00
Michael Wood 66b5608ffe bitbake: toaster: toastertable remove title from Show all in table
Title is often very long so this is not a great property to use here.

(Bitbake rev: 541ff2b5e9c6ebec0d0ced59b3f61cbd37bd37c2)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:18 +00:00
Michael Wood ce7289613b bitbake: toaster: Add recipe details page
This page provides the details of recipes, showing their packages and
additional metadata. It also allows you to build the recipe or customise
the recipe if it is an image recipe.

[YOCTO #8070]

(Bitbake rev: b6023619397f679791b7eff1faac06f4246133ce)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:17 +00:00
Michael Wood 5f52614a87 bitbake: toaster: newcustomimage Move modal dialog out of newcustomimage template
Move the modal template and JS out of the newcustomimage page so that it
can also be used by the image details page.

(Bitbake rev: c310bc6bab1a33124906dd57b3c63462a773ff25)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:17 +00:00
Michael Wood 2a3dd32d66 bitbake: toaster: Continue front end features to custom image recipe page.
Continuation of the work on the custom image recipe page, this brings
in:

- Basic notification of having added/removed a package.
- Connect up Build button
- Download recipe feature
- No packages states
- Project bread crumb
- Display additional recipe metadata
- Update accessors for recipe object inheritance changes

[YOCTO #8082]

(Bitbake rev: a656756a9255ec5882686ce9563d17f2eb3136e3)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:17 +00:00
Michael Wood d6e7e4ad43 bitbake: toaster: tables Add table for Packages and update SelectPackagesTable
Create a Packages table for use as the image details page.
Change the SelectPackagesTable table to inherit from the Packages table.
Remove the need for a separate view by adding the additional template
context items to the Table's page context.

(Bitbake rev: 336b1d8369d9e86ece78b63cb0e140e653216011)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:17 +00:00
Michael Wood 43f0a05fa4 bitbake: toaster: views Add view to download custom recipe
View to provide the custom recipe download feature. The recipe is
generated on-demand to make sure that it is the most current version of
the Custom recipe.

(Bitbake rev: 2101c854bb2d7ff1e3a4f00ad4d33d77859439ed)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:17 +00:00
Michael Wood 2cf55afb97 bitbake: toaster: move CustomImageRecipe generation to API entry point
Use the CustomImageRecipe generate_recipe_file_contents to generate the
recipe that we build from. Move creation of the dummy layer and recipe
object to the point of recipe creation as we need these objects before
the build time. Also update the methods to add and remove packages to
account for the CustomImageRecipe inheriting from Recipe.

(Bitbake rev: f3322567378d6038a00da0fab6c5641a1a8e5409)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:17 +00:00
Michael Wood c402ac2654 bitbake: toaster: orm add CustomImageRecipe generate contents function
Add function generate_recipe_file_contents to dump the custom image
recipe instance to a string for use either to push to the user as a
downloaded version of their custom image recipe or to use to generate
the recipe that we build.

(Bitbake rev: 6863343c3434ce19aa4b609c83f48a06e6943366)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:17 +00:00
Michael Wood a6e4f94b01 bitbake: toaster: buildinfohelper Add the concept of CustomImagePackage
This adds the concept of CustomImagePackage this is similar to the way
layers and recipes work in that we have a set of data which is part of
the build history and a set of data which is part of the configuration
data that toaster uses to guide people in configuring their project. We
create a set of  built_packages for every build but only create a package
for configuration purposes if we don't already have one, so that the
CustomImagePackage only ever contains a unique list of packages that are
available to be added and removed from a CustomImageRecipe.

(Bitbake rev: f81bb65883baa6c0f8a4d48a4de3291a10543992)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:17 +00:00
Michael Wood e1bfe1ceb6 bitbake: toaster: orm: Add db migration for new CustomImagePackage table
(Bitbake rev: c7da71fe8509439656f482c16ed081cf442f4030)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:17 +00:00
Michael Wood f760a78e3c bitbake: toaster: orm Add CustomImagePackage table
This table is used to track all the available packages in the current
toaster. Many of these packages belong to many CustomImageRecipes.

(Bitbake rev: c1bd4f760cd35535e44f488250e0a56b99cad680)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:17 +00:00
Michael Wood 4117af29c6 bitbake: toaster: orm: Add db migration for new CustomImageRecipe inheritance change
(Bitbake rev: 2831d74201abba68c301d85ee583d706a51d5a5f)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:16 +00:00
Michael Wood 1f102890d6 bitbake: toaster: orm make CustomImageRecipe inherit from Recipe
This allows us to re-use the properties of a recipe for the custom image
recipes as well as re-using the existing templates and logic that deals
with recipe objects.

(Bitbake rev: bb8120b56be7eee6ed2e4434d8477282a01e0c00)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:16 +00:00
Michael Wood 648753b30d bitbake: toaster: orm Add sum of dependencies size function to PackageDependencyManager
Add function that returns the Sum of the size of all the packages which depend on a package.
Access get_total_source_deps_size via a packages's dependency
manager.

(Bitbake rev: e4c86d2f6c2e86ad054b37d0a5bf7464a4de4f9a)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:16 +00:00
Michael Wood a92fc3025b bitbake: toaster: tablejs Add an event handler to manually trigger a data reload
Allow users of ToasterTable to manually trigger a refresh of the data.
This can be useful if an action has happened in-page and the data is now
invalid. Such as new data being added or removed from the model.

(Bitbake rev: 6e42070d8abc80dacd8094c4f5019577453a9d49)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:16 +00:00
Michael Wood 4c82878225 bitbake: toaster: ToasterTables simplify filter function move common part to widget
Move part of the functionality of the filter functions to the Table
widget. We don't need to implement it in each subclass.

(Bitbake rev: 16e48829f6fd96c1d21339253871f2a9b2446f87)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:16 +00:00
Michael Wood 3e1e8e6a99 bitbake: toaster: models fall back to a sensible string for no vcs reference
Fall back to a 'n/a' string for the vcs reference, not all our source
has to be in a vcs and therefore it is legitimate for this to be none.

(Bitbake rev: 1739b509e9efc4b016fc73c2d4399f1f9d3d285f)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:16 +00:00
Michael Wood 14d09c8b22 bitbake: toaster: localhostbecontroller CustomRecipe now base_recipe is Recipe
Update the reference to the base_recipe. It is now a Recipe object
rather than an intermediate AvailableRecipe object.
Therefore doesn't need an extra traverse down the object hierarchy.

(Bitbake rev: 8056ec65bd93005ecb7b0ed12dcb21b3b60df22c)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:16 +00:00
Aníbal Limón d0d85a4d6b bitbake: bb/fetch2: Move export_proxies function from wget to utils.
In order to use in other modules since is a common function
when needs to get proxies working.

(Bitbake rev: 85c529044381895556d603a3974de22392646a22)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-08 10:52:40 +00:00
Elliot Smith 6228cf8166 bitbake: toaster: reinstate ID on edit columns button
The ID was omitted from the "edit columns" button on the
"projects", "all builds" and "project builds" tables when they
were converted to ToasterTable. This caused the QA tests to fail.

Reinstate the ID so the tests can identify the "edit columns" button
correctly.

[YOCTO #9051]

(Bitbake rev: 934b82badcf063c8ff252d806c2fb019f7a2e55f)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-06 23:13:47 +00:00
Ed Bartosh 916c73dfef bitbake: cooker: shutdown cooker parser on shutdown
Currently any not catched exception in cooker causes bitbake
to hang because of not terminated children of CookerParser.

Long term solution would be to reimplement Cooker as a context
manager and terminate parser children in its __exit__ method.

Partial fix is to call CookerParser.shutdown in Cooker.shutdown in
hope that all Cooker exceptions are caught and shutdown method is
called.

[YOCTO #8900]

(Bitbake rev: 3f67600dc3292bc8208644ce89e8bf7ab95cf2e7)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-06 23:13:47 +00:00
Richard Purdie 8857498ba4 bitbake: fetch2/osc: Clean up old variable syntax
Since people do copy and paste these things, clean up old syntax styles.

(Bitbake rev: 4fb028b0bd14d3e4b3fd7a89c643528728566476)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-06 23:13:47 +00:00
Richard Purdie 54da82972c bitbake: fetch2/osc: Remove hardcoded url
This shouldn't be in here, use a variable instead.

(Bitbake rev: 2e25d09a1ab62ccc3573d13114d59838cf4b07f2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-06 23:13:47 +00:00
Richard Purdie 6ae662acdc bitbake: parse/ast: Mark anonymous functions as python functions
Anonymous functions are python functions, set the variable
flags as such so we can detect them and avoid expansion where
needed.

(Bitbake rev: 1b303785c578bbae3a89be8d751d80fba860f62e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-04 23:47:49 +00:00
Richard Purdie 9913fd88d9 bitbake: codeparser: Improve handling of data.expand() dependencies
Currently bitbake doesn't parse into data.expand() expressions,
relying on high level expansion of python code to handle this.

One of the tests does however test this works.

We don't really want to be doing string expansion on python code,
so specifically parse into expand() function calls so that when
the high level behaviour is tweaked, the self tests continue to
pass and that we do continue to handle expand() function calls as
best we can.

(Bitbake rev: b12c17be5e4a74c9680876605c87f46501f78d28)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-04 23:47:49 +00:00
Richard Purdie 4628fe12e7 bitbake: lib/bb: Add expansion parameter to getVarFlag
This sets the scene for removing the default False for expansion from
getVarFlag. This would later allow True to become the expand default.

On the most part this is an automatic translation with:

sed -e 's:\(\.getVarFlag([^,()]*, [^,()]*\)):\1, False):g'  -i `grep -ril getVar *`

There should be no functional change from this patch.

(Bitbake rev: 7c3b99c6a716095af3ffce0b15110e91fb49c913)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-04 23:47:49 +00:00
Felipe F. Tonello b98866d003 bitbake: fetch2/gitsm: Fix when repository change submodules
This fix a problem when checking out a commit that changes the submodules
previously checkout.

Example:
Recipe uses branch A and then it updates to use branch B, but branch B has
different submodules dependencies then what branch A previously had.

(Bitbake rev: 54a3864246f2be0b62761f639a1d5c9407aded4f)

Signed-off-by: Felipe F. Tonello <eu@felipetonello.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-04 23:47:49 +00:00
Richard Purdie 390c2c1268 bitbake: data_smart: Add missing expand parameter to getVar call
(Bitbake rev: 2876019e696ff2af164961d5d4c8e3ea9dfefc23)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-04 23:47:49 +00:00
Diego Santa Cruz 56454f6fb2 bitbake: bitbake: prserv: do not clear umask when daemonizing
Clearing the umask when daemonizing is not the correct thing
to do, as it will create files writable by anyone by default.
For instance the pid file was being created with mode 777.
This could also potentially affect the sqlite database.
Better let the calling process decide on the umask.

[YOCTO #9036]

(Bitbake rev: ff6d3f53a4504eae7ec4c190b9f7595b09aed017)

Signed-off-by: Diego Santa Cruz <Diego.SantaCruz@spinetix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-04 23:47:48 +00:00
Diego Santa Cruz abf8a8fbc3 bitbake: bitbake: prserv: SIGTERM handling hung process
The current SIGTERM handler hungs the process instead of
making it exit. The problem seems to be that the handler thread
is not signaled to quit, so it stays there doing its work, as
it is not a daemon thread. Setting the quit variable fixes this.

While at it, to not use the SystemExit exception to terminate
upon SIGTERM but instead left the quit flag do its job. This way
the PID file is properly removed.

[YOCTO #9035]

(Bitbake rev: 655ec800d54da581229f12efb6f0baf54975fed4)

Signed-off-by: Diego Santa Cruz <Diego.SantaCruz@spinetix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-04 23:47:48 +00:00
Diego Santa Cruz be032fc40e bitbake: bitbake: prserv: -wal and -shm sqlite lost when daemonizing
When daemonizing the PR service the -wal and -shm sqlite files were
being deleted, although they should never be. While the daemonized
process keeps the file descriptors open and thus a clean exit does
not loose any data, a power outage would loose all data in the WAL.
Removing these files also breaks sqlite collaboration between
processes and furthermore prevents taking proper backups without
stopping the PR service.

The reason this happens is that the DB connection is opened in
the initial process, before forking for daemonization. When the
DB connection is closed by the exiting parent processes it can
delete the -wal and -shm files if it considers itself to be the
last connection to the DB. This is easily fixed by opening the
DB connection after all forking.

[YOCTO #9034]

(Bitbake rev: bc867c81e3894da5bdd2e45fa695bb5f5f1bb58b)

Signed-off-by: Diego Santa Cruz <Diego.SantaCruz@spinetix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-04 23:47:48 +00:00
Phil Blundell a4a5d1f323 bitbake: cooker, bitbake-worker: Fix spelling of "received"
I before E, except after C...

(Bitbake rev: 14c9593265f7469cb8a205a46f845ac7491246df)

Signed-off-by: Phil Blundell <pb@pbcl.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-01 11:20:42 +00:00
Richard Purdie 8f6b9c75a7 bitbake: cooker: Only start as many parse threads as we need
If we're only going to parse one recipe, no point in starting
a large number of threads.

(Bitbake rev: b977faf59dc08050a44a16032fe52d1bbb80f2a1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-01 11:20:42 +00:00
Richard Purdie 602da7c921 bitbake: knotty: Don't show errors for universe provider issues
When running universe builds, we don't expect an error exit code for
provider warnings. Change the error messages to warnings in this case.

This deals with errors causing problems on our autobuilders amongst
other issues.

(Bitbake rev: d4989fb0355476de172169f0698757f7360e9a1f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-01 11:20:42 +00:00
Sujith H 3072361f86 bitbake: bitbake: BBUIHelper: Remove function findServerDetails
findServerDetails function can be removed safely
from the source tree. Couldn't find any files
calling this function.

(Bitbake rev: 46871f769db13ccd36deedc5b6f3dbc0a3d31c4b)

Signed-off-by: Sujith H <sujith.h@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-30 11:43:56 +00:00
Peter Kjellerstedt 28c041c1e7 bitbake: fetch2: Simplify logic in verify_checksum()
The recent change to verify_checksum() to only show checksum warnings
if no checksums are supplied made it possible to simplify the logic a
bit more.

(Bitbake rev: 1dc00b874acae44bbba9d8028d94f7bc97ddcd76)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-30 11:43:56 +00:00
Richard Purdie 5375e6431c bitbake: bitbake: Set process names to be meaninful
This means that when you view the process tree, the processes
have meaningful names, aiding debugging:

$ pstree -p 30021
bash(30021)───KnottyUI(115579)───Cooker(115590)─┬─PRServ(115592)───{PRServ Handler}(115593)
                                                ├─Worker(115630)───bash:sleep(115631)───run.do_sleep.11(115633)───sleep(115634)
                                                └─{ProcessEQueue}(115591)

$ pstree -p 30021
bash(30021)───KnottyUI(117319)───Cooker(117330)─┬─Cooker(117335)
                                                ├─PRServ(117332)───{PRServ Handler}(117333)
                                                ├─Parser-1:2(117336)
                                                └─{ProcessEQueue}(117331)

Applies to parse threads, PR Server, cooker, the workers and execution
threads, working within the 16 character limit as best we can.

Needed to tweak the bitbake-worker magic values to tell the
workers apart.

(Bitbake rev: 539726a3b2202249a3f148d99e08909cb61902a5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-30 11:43:56 +00:00
Richard Purdie 5b234d1539 bitbake: utils: Add ability to change the process name
Being able to tell the bitbake processes apart is useful for debugging.
Add a helper function which allows this without making it a hard
dependency. Errors are ignored, this is just nice to have.

(Bitbake rev: fd7f1a94d196b8a3c445e313d9e699b352b1da97)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-30 11:43:55 +00:00
Andre McCurdy 0b06924759 bitbake: data.py: avoid double newlines at the end of functions in emit_var()
A newline is always appended to the function body when it's written
out, so strip any trailing newlines which may be there already.

(Bitbake rev: 8a3f50936113e15d2f2822f6aee494204fa1c24f)

Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-30 11:43:55 +00:00
Andre McCurdy 68600aee2d bitbake: build.py: minor shell_trap_code() formatting tweaks
Fix quoting of $BASH_COMMAND and avoid wrapping at 80 columns (the
script which follows is likely to contain some very long lines, so
line wrapping in bb_exit_handler() looks somewhat out of place).

(Bitbake rev: 8e12c8f8441a7c6a03e603c5789d6037945704c1)

Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-30 11:43:55 +00:00
Peter Kjellerstedt d61d290ad7 bitbake: bitbake-user-manual-ref-variables: Update the help for BBMASK
Update the help for BBMASK to reflect that it is now allowed to
contain multiple regular expressions. Also changed the examples for
BBMASK to be a bit more diverse.

(Bitbake rev: 023a5b69df05adce0ed192b393b45eb879c096a0)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-29 17:03:53 +00:00
Peter Kjellerstedt a948f5252a bitbake: cooker: Allow BBMASK to contain multiple regular expressions
Before, BBMASK was only permitted to contain one regular expression.
This made it hard to add to the BBMASK in multiple places as one was
supposed to separate the different regular expressions with a "|"
rather than with whitespace as is customary in BitBake variables.

Now one can specify any number of regular expressions in BBMASK. This
makes it possible to, e.g., mask out recipes in another layer from the
layer.conf file.

This also properly ignores any regular expressions that do not compile
(before an invalid regular expression would cause a ParseError in the
first bbappend file found stating that it was not a BitBake file...)

(Bitbake rev: 2c778ad50aceaffb855baf5f4aa0fed98c880870)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-29 17:03:53 +00:00
Martin Samuelsson e82101a4ed bitbake: bitbake-user-manual-metadata: Updated 'dir' flag
Added more detail to the 'dir' flag to state the last
directory listed will be used as the work directory for the
task.

(Bitbake rev: 2fbd9e4e44f6b491cccca81fd34586fcfdeea3d3)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-29 17:03:53 +00:00
Scott Rifenbark 100d6c2782 bitbake: bitbake-user-manual: Updated the example BitBake directory
This directory no longer has setup.py as part of the directory.
I removed that line from the example in the
"Obtaining BitBake" section.

Fixes YOCTO #8623

(Bitbake rev: 6a82f6432022debe48830db98b3fb16424411a87)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-29 17:03:53 +00:00
Ross Burton 9f22898e0f bitbake: fetch2/wget: fallback to GET if HEAD is rejected in checkstatus()
The core change here is to fall back to GET requests if HEAD is rejected in the
checkstatus() method, as you can't do a HEAD on Amazon S3 (used by Github
archives).  This meant removing the monkey patch that the default method was GET
and adding a fixed redirect handler that doesn't reset to GET.

Also, change the way the opener is constructed from an if/elif cluster to a
conditionally constructed list.

(Bitbake rev: 6ec70d5d2e330b41b932b0a655b838a5f37df01e)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-26 22:34:45 +00:00
Daniel Istrate 72f98ba577 bitbake: toaster: Update UI test runner
Add new runner options:
    --run-all-tests: finds all tests, ignores config
    --run-suite <suite> (from cfg)

Without arguments, run tests from current os section (config), e.g.:
    1. ./run_toastertests
    2. ./run_toastertests --run-all-tests
    3. ./run_toastertests --run-suite darwin

Update toaster logging to meet QA CI requirements.

(Bitbake rev: 5685feb51fbb6d54fde6027cc765b9edd8eda65a)

Signed-off-by: Daniel Istrate <daniel.alexandrux.istrate@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-25 16:29:10 +00:00
Richard Purdie 4a5458f38f bitbake: fetch2: Don't show checksum warnings if a single checksum was supplied
If one checksum is supplied to a SRC_URI, we really don't want to show
warnings about the other type which isn't present as one checksum
is really good enough for most cases.

(Bitbake rev: 43358a9b595b2928458a5f463cf1949394160c3a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-24 09:40:42 +00:00
Ed Bartosh bd1a534a11 bitbake: toaster: run bitbake server with --read option
Toaster script run bitbake with --postread conf/toaster.conf.
It turned out that variables set this way don't influence the
build. Changing --postread to --read makes bitbake to read
variables earlier and should fix this issue.

[YOCTO #8781]

(Bitbake rev: 76c9871740ef42ac35fdfdcb89a68478cca370cd)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-22 12:45:44 +00:00
Paul Eggleton 76a281c870 bitbake: taskdata: add the ability to access world targets list
In certain circumstances it can be useful to get access to the world
targets list from a recipe in order to add dependencies on some or all
of the items in it. If a special function, 'calculate_extra_depends' is
defined in the recipe, and the recipe is to be built, then call it at
the right point before we calculate which tasks should be run. The
function can append items to the "deps" list in order to add
dependencies. This is not as tidy a solution as I would have liked, but
it does at least do the job.

As part of this change, the buildWorldTargets function was moved to
bb.providers to make it possible to call from taskdata.

Part of the implementation of [YOCTO #8600].

(Bitbake rev: aba0dce57c889495ec5c13919991a060aeff65d2)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-22 12:45:44 +00:00
Robert Yang 11a1f49630 bitbake: cache.py: check existence before add to cachedata.rproviders
The rprovides maybe contain duplicated lines when parse again, we need
check it before add to cachedata.rproviders, similar to what we had done
to cachedata.providers.

(Bitbake rev: 6c488afb0fe30a9655ec62a1d22f9f388365f012)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-22 12:45:44 +00:00
Robert Yang 05c17750c3 bitbake: taskdata.py: add RuntimeProviders to close matches
This is useful for newbie, for example:
$ bitbake rpm-build
ERROR: Nothing PROVIDES 'rpm-build'. Close matches:
  pm-utils
  rpm RPROVIDES rpm-build

[YOCTO #8881]

(Bitbake rev: 4b59eb8cc2321fe72f2988b6c9c0fecd4883255b)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-22 12:45:44 +00:00
Richard Purdie cf9cb65eec bitbake: data_smart: Don't show exceptions for EOL literals
If variables are unset, the code simply doesn't expand them, there
aren't errors. If the code is a python expression, this can get a bit
messy, see the attached test case. The python expansion code sees the }
of the unexpanded value rather than the close of the python expression
and then raises a SyntaxError exception.

Ideally, we'd update the code to match pairs of brackets. I don't know
how to do that with the current regex and this is unfortunately a
performance sensitive piece of code. We also run the risk of breaking
existing code in OE-Core where there are "{" characters but not "}"
to close them (PKGE and PE).

Rather than raising the exception, matching the existing "just return
the expression" behaviour seems more consistent with the standard
variable behaviour.

This addresses an issue found in the recent image.bbclass code where
there are some variables we choose not to expand (TMPDIR/DATETIME).

This patch also adds a test case for this behaviour. It wouldn't preclude
improved bracket matching code in the future either.

(Bitbake rev: d80d39e73223a50fda0090784303d2c57167bb4c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-22 12:01:24 +00:00
Olof Johansson c886a786cf bitbake: tests/codeparser.py: Add filename/lineno flags to test variable
A recent change in bitbake added filename/lineno information to the
parameters of bb.data.build_dependencies(). The codeparser tests
required a little adaption to the changes, adding the flags to the FOO
variable used in the tests.

The error seen when running the tests is a TypeError exception raised
in bb.codeparser:

  TypeError: int() argument must be a string or a number, not 'NoneType'

(Bitbake rev: f1fe674397ac5cd355696d5b4cc90b7cfa6c867f)

Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-19 17:38:51 +00:00
Ed Bartosh f130033856 bitbake: toaster: write variables to toaster.conf
With the resent change in rootfs creation code setVariable
rpc calls don't set variables for bitbake workers anymore.

Writen variables to toaster.conf should solve this issue.

[YOCTO #8910]

(Bitbake rev: d6dfe40320ff6ca420d9f9016b4d1d83d10f1d59)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-19 17:38:51 +00:00
Richard Purdie f0d47a6d9a bitbake: Revert "runqueue.py: Ensure one setscene function doesn't mask out another which needs to run"
This reverts commit b22592af81.

That commit isn't entirely clear about why this change is needed but
I do have a usecase where this breaks things. If for example you run
"bitbake X -c packagedata" and that packagedata is in sstate, you'd
expect this to work.

If sstate doesn't contain a do_populate_sysroot for a dependency, you
would still expect the command above to succeed and you would not
expect
it to rebuild that dependency. With the current code, this isn't what
happens. The code finds the sstate for do_populate_sysroot missing,
this makes the task "uncovered" and this in turn makes it unskippable.

The example I found with this was avahi-ui, where it would trigger
a build of libdaemon to obtain its populate_sysroot.

Since this behaviour seems completely incorrect, revert the older patch
and we'll address any issues that crop up as a result.

(Bitbake rev: 36a9840a5da17cc14561881fdd6a4f2cb0a75e49)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-19 17:26:15 +00:00
Elliot Smith f98e11c809 bitbake: toastergui: make artifact download more robust
When an artifact download is requested, Toaster goes through a
convoluted series of conditions to decide which file to push
to the response. In the case of build artifact downloads for
command line builds, this caused an ugly exception, as command
line builds don't have a build request.

To simplify and catch more corner cases, remove the code which
fetches files via the build environment (we only support the local
build environment anyway). Then push all requests along a single
path, catching any missing file errors, missing object errors
or poorly-formed URLs in a single except clause which always returns
a valid response.

Also modify the text on the "unavailable artifact" page so it
says that the artifact doesn't exist, rather than it "no longer"
exists (exceptions may occur because an invalid artifact was
requested, rather than an artifact which was removed).

[YOCTO #7603]

(Bitbake rev: 24e20db55c2933de5e58ca754b8fd5b624f47820)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-18 22:06:46 +00:00
Elliot Smith 68f3e1e067 bitbake: toasterui: log OSErrorException metadata events
OSErrors occurring in toaster.bbclass are converted to
OSErrorException metadata events. They were then being swallowed
as unprocessed events by toasterui, which made them difficult
to spot.

Explicitly catch OSErrorException events and log them so they
are easier to spot and debug.

(Bitbake rev: 69f2b2bc373ce114609600b59a6b6ccef20771c9)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-18 22:06:46 +00:00
Elliot Smith fb94754e83 bitbake: toasterui: listen for bb.event.MetadataEvent
The event mask for toasterui doesn't include MetadataEvents.
This means that we're missing the ArtifactFileSize event
(among others), which is the one we use to populate the SDK
artifact table.

Add that event type to the toasterui event mask so we can
record SDK artifacts as they are created.

[YOCTO #7603]

(Bitbake rev: d0276a831bb8cffd42c8367895633eaa1fa1ed30)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-18 22:06:46 +00:00
Ed Bartosh 0fe6e2dc7b bitbake: toaster: increase timeout
Sometimes Toaster connection to the bitbake server fails with
this error in the log:
   ERROR: Could not connect to server 0.0.0.0:46572
   : Could not register UI event handler
It happens more often on slow machines.

Increasing timeout after restarting bitbake should fix this issue.

[YOCTO #8776]

(Bitbake rev: b64a0adc1efd72aa4d46df3e043dd14543abae3d)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-16 10:36:50 +00:00
Elliot Smith 798e8b8d03 bitbake: toastergui: code formatting and clean-up
Minor fixes to code formatting and small improvements from
code review.

(Bitbake rev: 2c97f3a5c6ae37de910deb90390c5b856a600c5f)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-15 16:30:01 +00:00
Elliot Smith c4b50111e9 bitbake: toaster tests: fix Django tests for new ToasterTable pages
The Django command-line tests can no longer test the content
of the projects/, builds/ and projectbuilds/ pages, as
ToasterTable pages are populated by JavaScript.

Fix/remove affected tests by converting them to tests on the
JSON returned by the ToasterTable.

[YOCTO #8738]

(Bitbake rev: 85efa9530fa6181855e051bfd14de1c15db9c3b7)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-15 16:30:01 +00:00
Elliot Smith 88a262cbd2 bitbake: toastergui: remove unused views and template code
The code in views.py for setting up the template context for
old non-ToasterTable views is no longer necessary, as this
is now implemented in tables.py.

The template files for these views have also been removed.

[YOCTO #8738]

(Bitbake rev: 2b5a13afb068c85466436914d8d4ac3b31bc5c02)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-15 16:30:01 +00:00
Elliot Smith 059a274aa9 bitbake: toastergui: fix error and warning counts for builds
The error and warning counts displayed for builds were counts of
the errors and warnings objects associated with a build. Because these
values were being derived on the fly, it was not possible to sort by
them.

Previously, the 3rd party django-aggregate-if library was used to
add aggregate fields to Build objects and should then have been
used to populate the "all builds" and "project builds" tables. However,
at some point the templates had changed so that the error and warning
counts were coming from the properties on the Build model and not from
these aggregates. This meant that it was not possible to sort by these
fields.

Django 1.8 supports conditional aggregates in annotation fields on
querysets. This means we can remove django-aggregate-if, use the new Django
1.8 feature to derive errors_no and warnings_no fields as annotations,
then use those annotation fields in the templates. This makes the "builds"
tables sortable again.

[YOCTO #8738]

(Bitbake rev: 9be7c5c18b325f6ed40bc431ac408db242007eb1)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-15 16:30:01 +00:00
Elliot Smith 4103e0cb74 bitbake: toastergui: make "Apply" button state depend on filter range
If a range filter action had an empty from/to field, the range
filter could still be applied. This was confusing, as an invalid
filter range caused all records to display, even though a filter
appeared to have been applied (by the highlighted state of
the filter button).

Change the state of the "Apply" button, disabling it if the radio
button for a range filter action is selected but the range is
incomplete (from or to field is empty).

When a non-range filter is selected, the "Apply" button always
enable the "Apply" button.

[YOCTO #8738]

(Bitbake rev: 168184b28165d7aa354b9092b5986f91c58d550d)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-15 16:30:00 +00:00
Elliot Smith 6c2d88fda5 bitbake: toastergui: mute label for filter actions with no records
The radio button for a filter action is disabled if that filter
action has no associated records. However, the label retains
the normal font styling, so it's unclear that the action is
not available.

Add the "muted" class to the label for a filter action (and still
disable its radio button) if it has no records associated with it.

[YOCTO #8738]

(Bitbake rev: b7f7ff095c9c4c922e608f776713f17acc1f150d)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-15 16:30:00 +00:00
Elliot Smith f08730ab4d bitbake: toastergui: set default visible and hideable columns
Incorrect columns were shown by default in the "all builds",
"project builds" and "all projects" pages.

Set the "hidden" property on columns in these tables to hide the
correct columns.

Add a set_column_hidden() method to ToasterTable so that the
"hidden" property can be overridden for the machines column
in the project builds page (it shares a superclass with
all builds).

Make the time column on all builds page hideable.

[YOCTO #8738]

(Bitbake rev: be3982c71703eaa51e7f3352e0cb5b3af11c9ead)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-15 16:30:00 +00:00
Elliot Smith 112f3746cd bitbake: toastergui: serialise decimals correctly
The conversion of some ToasterTable Build object querysets to
JSON caused a serialisation error. This is because one of the
fields in the queryset was of type decimal.Decimal, and our
serialiser didn't know what to do with it.

Add a clause to check for decimal fields and serialise them
so that correct JSON can be generated.

(Bitbake rev: fa6229d4edf5904ccaa9dc323d0ab2318d1ef314)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-15 16:30:00 +00:00
Elliot Smith e024aab39c bitbake: toastergui: streamline construction of filter objects
In line with comments from review, remove the QuerysetFilter
class (redundant) and convert ProjectFilters into a class
with static methods.

[YOCTO #8738]

(Bitbake rev: 59379bf6467029223045c5ebef868729d8e02c86)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-15 16:30:00 +00:00
Elliot Smith fcb20f9dfd bitbake: toastergui: ensure filter_value updates
Clicking on the radio button for a date range filter action
populates the from and to fields for that action if they are empty.

However, because this doesn't fire "change" events, clicking on
the radio button doesn't update the filter_value hidden field. This
means that the date range action's filter_value parameter isn't
set correctly when the filter popup is submitted.

Manually call the changeHandler() to set the filter_value whenever
the radio for a date range filter is clicked.

[YOCTO #8738]

(Bitbake rev: 1a3038cf8d9b32532f1fe92cd3472b4473ffc0c4)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-15 16:30:00 +00:00
Elliot Smith f9c46f5e5e bitbake: toastergui: don't hide all elements with .col class
Disabling the "project" column in a ToasterTable for builds
causes the recent builds area to be hidden. This is because
the column hiding code hides all elements with a class matching
".<column>", regardless of where they occur on the page; and
the recent builds area was using the ".project-name" class,
which means it is included in the set of elements which are hidden.

Scope the element search to the table so that only elements
within the table are hidden or shown.

[YOCTO #8792]

(Bitbake rev: 151bc20daf51e0ab8baf11ea29244b7fec1f8a22)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-15 16:30:00 +00:00
Elliot Smith eaae82a19a bitbake: toastergui: convert project builds page to ToasterTable
Use the all builds ToasterTable as the basis for the project builds
ToasterTable.

[YOCTO #8738]

(Bitbake rev: 87bcfb740dd2d9944e35a2a1f71cbf8ff3b266e9)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-15 16:30:00 +00:00
Elliot Smith 33b011c158 bitbake: toastergui: implement "today" and "yesterday" filters
Add the "today" and "yesterday" filters to the started_on
and completed_on columns in the builds table.

During this work, some minor adjustments were made to the
behaviour of the builds table:

* Amend filter action variable names so they're more succinct.
* Retain order in which actions are added to a filter, as this
ordering is used in the UI when displaying the filter actions.
* Always show the table chrome, otherwise it's not possible
to edit the columns shown until there are 10 or more results.
* Because date range searches may return no results, make sure
that the search bar and "show all results" link are visible
when the query returns no results.

[YOCTO #8738]

(Bitbake rev: f17cfa009e58833e0e55884fa04de8abd522b6bc)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-15 16:30:00 +00:00
Elliot Smith f8d383d87f bitbake: toastergui: implement date range filters for builds
Implement the completed_on and started_on filtering for
builds.

Also separate the name of a filter ("filter" in the querystring)
from its value ("filter_value" in the querystring). This enables
filtering to be defined in the querystring more intuitively,
and also makes it easier to add other types of filter (e.g.
by day).

[YOCTO #8738]

(Bitbake rev: d47c32e88c2d4a423f4d94d49759e557f425a539)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-15 16:30:00 +00:00
Elliot Smith b929889cdd bitbake: toastergui: show recent builds on all builds page
The recent builds section was disabled while converting the
all builds page to ToasterTable.

Re-enable the recent builds area and add the data it requires
to the ToasterTable context.

[YOCTO #8738]

(Bitbake rev: d6df4545bd134a23c9bd3cd1ba3b61ddb26545e4)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-15 16:30:00 +00:00
Elliot Smith 1a4b203ca5 bitbake: toastergui: switch off filter highlights when inactive
In ToasterTables with multiple columns which allow filtering
(e.g. all builds), selecting one filter, then a second filter
(e.g. selecting "failed builds" then "outcome" for all builds),
would result in both filters being highlighted at the same time.

Fix this by removing the "active" highlight on all column filter
buttons when a new filter value is submitted (via the filter modal).

NB to enable this, added a data-filter-on attribute to all
column filter buttons to make them easy to select.

[YOCTO #8738]

(Bitbake rev: 7347ad0d4baace593751b44a86ab8e11a04a02b6)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-15 16:29:59 +00:00
Elliot Smith 809046c6fb bitbake: toastergui: refactor ToasterTable filtering
The filter code for ToasterTable was difficult to follow
and inflexible (not allowing different types of filter, for example).

Refactor to a set of filter classes to make the structure cleaner
and provide the flexibility needed for other filter types
(e.g. date range filter).

[YOCTO #8738]

(Bitbake rev: 94031bb30bdaf665d0c8c68b591fcb7a17b6674d)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-15 16:29:59 +00:00
Elliot Smith 294579b531 bitbake: toastergui: convert all builds page to ToasterTable
For better long-term maintainability, use ToasterTable instead
of Django template and view code to display the all builds page.

NB the builds.html template has been left in, as this will
otherwise cause conflicts when merging the new theme.

[YOCTO #8738]

(Bitbake rev: e0590fc8103afeb4c5e613a826057555c8193d59)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-15 16:29:59 +00:00
Elliot Smith 6c12ca7f93 bitbake: toastergui: use event delegates for hover help elements
libtoaster.js binds to hover help elements via their hover() and
mouseout() methods. However, any elements added to the DOM after
libtoaster has initialised will not have these bindings added.
This causes a problem for ToasterTables which have hover-help
elements (e.g. the builds/ table).

Use the on() method instead. This uses event delegation to bind
a handler to any th or td elements already in the DOM, or
which will be added to the DOM in future. ToasterTables can
now reconstruct the table DOM and still have the correct handlers
attached once the table is done.

[YOCTO #8738]

(Bitbake rev: 1f3ff01fed0b4de8721191f108033ad044cdc26a)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-15 16:29:59 +00:00
Elliot Smith ef93dce7e1 bitbake: toastergui: switch projects/ view to ToasterTable
Remove the old projects page and replace with the new
ToasterTable-based version.

NB although the projects.html template is no longer required,
it's been left in as there will be changes applied to it for
the new theme. These changes will have to then be transferred
from the projects.html template to projects-toastertable.html.

Similarly, the code for the projects page in views.py has been
retained.

[YOCTO #8738]

(Bitbake rev: ebe7831ef65e78a9d100f29a63311518577fc838)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-15 16:29:59 +00:00
Elliot Smith 417f1d3f53 bitbake: toaster: check inferred file suffixes against list of known types
The algorithm for finding the suffix for image files produced by
the build doesn't reference a list of known file suffixes, so
could be prone to error.

Modify how file suffixes are parsed from the file path so that
they are compared against a list of known types; if this fails,
use the part of the basename of the file path after the first
'.' character.

Also rationalise the places in the views code where we
extract the file name extensions for builds, so they both use
the same algorithm (before, the same code was duplicated in
two places).

[YOCTO #8417]

(Bitbake rev: dd1c509696b8ab5e593cc64637060a58e95fcd1f)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-15 16:29:59 +00:00
Elliot Smith c02ee05857 bitbake: toaster: move image file suffix list to model
Image file suffixes are used in the project configuration page to
show a list of available image file types. This list is stored
as a function in the views code.

However, this list is also needed when parsing image file paths,
so that the suffixes can be shown in the "all builds" and "project
builds" tables.

Move the list of valid image file suffixes to the Target_Image_File
class to make is accessible in other places where it may be needed.

[YOCTO #8738]

(Bitbake rev: c2f20232077917552623fd0726d0820e50b04cae)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-15 16:29:59 +00:00
Elliot Smith d29e4cd040 bitbake: toastergui: use ToasterTable for projects page
The projects page uses the old approach for showing tables,
which means a template for each table. This means that applying
changes to ToasterTable (which is used for most tables) has
no effect on the layout, styling and behaviour for these older
tables, and requires additional duplicated effort.

Move the projects page to use ToasterTable instead, to remove
the duplication of effort.

[YOCTO #8738]

(Bitbake rev: df56a35bc71639457329c4b58839976c5ee40106)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-15 16:29:59 +00:00
Richard Purdie f8508deb01 bitbake: Revert "fetch/git: Change to use clearer ssh url syntax for broken servers"
This reverts commit 4193e99adce8e88f12ac88d7578ad39575f7e346.

It seems the underlying issue was caused by ":" in the url which isn't
supported. The patch was therefore incorrect.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-15 15:02:40 +00:00
Richard Purdie da43a56d35 bitbake: Revert "fetch2/local.py: avoid using PREMIRROR"
This reverts commit e130dca85bac82bd4d88f94a6bf9fe36e8ad4d7c.

This is in fact a valid use case, for example the sstate.bbclass code
sets up SSTATE_MIRRORS as PREMIRRORS. Its quite common to map those
file:// urls to remote http:// urls and with the above change, this
no longer works.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-13 09:47:28 +00:00
Kristian Amlie 1724ffd32d bitbake: fetch2/git.py: Add missing "errno" module import.
This goes undetected most of the time, but when updating a repository,
if the ud.fullmirror file is not present, you end up getting an
exception instead of carrying on because the errno module is not
loaded (specifically "if exc.errno != errno.ENOENT").

(Bitbake rev: e6fca8480731ce817df9bee61438347a5e3d3017)

Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:28:55 +00:00
Chen Qi 74fa824e6c bitbake: bitbake: clean up stamp-base related codes
The 'stamp-base' and 'stamp-base-clean' related codes are no longer useful,
clean them up.

[YOCTO #8468]

(Bitbake rev: 7b4c42b315d4a26dd8f2ceb874a94737bf9f183e)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:28:55 +00:00
Richard Purdie 43328feccc bitbake: runqueue: Fix setscene task dependencies
Debugging suggests that setscene tasks are being a little greedy about their
dependencies, for example, lsof is insisting that gcc-runtime's do_package
is installed. If it isn't, its requiring gcc to rebuild.

If gcc-runtime do_package_write_xxx and do_packagedata is available, there
is no reason do_package should be needed.

The reason this is happening appears to be from the batching up of task
dependencies code, rather than setscene tasks stopping when passing over
a setscene task, they were being carried forward. This patch fixes it
so the data is 'zeroed' when passing over a setscene task boundary,
which gives the dependency graph that is expected.

After this patch, lsof will rebuild quite happily without
gcc-runtime:do_package being present, as expected. This should lead to
less dependencies being installed for builds from sstate and generally
better performance in general.

(Bitbake rev: f8bcb0a1e3b008b71c9a7cd21f76d0906f2d8068)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:26:33 +00:00
Michael Wood 7b905caf60 bitbake: toaster: settings Add uid to the toaster cache dir
Make the default toaster cache dir unique to the user running
toaster. If we have multiple users running toaster we previously
got a permission denied exception on saving a cache file.

[YOCTO #8782]

(Bitbake rev: 5207abdf58019271bf92bff4bcce3911b8691508)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:26:33 +00:00
Ed Bartosh dff7a277e3 bitbake: toaster: show 'satisfied via' text for reverse deps
Showed '<dependency> satisfied via <provider>' text and
help tooltip for the reverse build dependencies provided
through 'PROVIDES' in the 'Reverse build dependencies' tab.

[YOCTO #6169]

(Bitbake rev: c7bb98e2e2111790ded86087b13c8b49462d6b75)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:26:33 +00:00
Ed Bartosh 89f493261a bitbake: toaster: show 'satisfied via' text for build deps
Showed '<dependency> satisfied via <provider>' text and
help tooltip for the build dependencies provided through
'PROVIDES' in the 'Build dependencies' tab.

[YOCTO #6169]

(Bitbake rev: de77e338fe70341fe98561e2e40b534f5c88db10)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:26:33 +00:00
Ed Bartosh febb898a06 bitbake: toaster: show list of provides for the recipe
Showed list of names that recipe provides.

(Bitbake rev: 60318c9a049292bd33322d8446a629d778337e8a)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:26:33 +00:00
Ed Bartosh 2ff4ccb13f bitbake: buildinfohelper: add provides info to the db
Added new entries to Provides model and link them to
Recipe_Dependency using 'via' field.

This data will be used by Toaster UI to show 'Provides:'
information for the recipes.

[YOCTO #6169]

(Bitbake rev: 336ddc8df611d4c8f1c3d3a06d0a85bb544c38bc)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:26:32 +00:00
Ed Bartosh 16a81fb97a bitbake: toaster: add Provider model
Added new model Provider and a foreign key 'via' to link
Recipe_Dependency to it.

[YOCTO #6169]

(Bitbake rev: e45fff6314741d46e2549b2f72ed380cbbb95593)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:26:32 +00:00
Ed Bartosh 6a28ed33b7 bitbake: buildinfohelper: use providermap
Used providermap in store_dependency_information function
to find virtual dependencies. This should fix annoying
warnings "stpd: KeyError saving recipe dependency"

[YOCTO #6169]

(Bitbake rev: 85c416ca338c886db6e79651e44727482df9fb07)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:26:32 +00:00
Ed Bartosh f2b7252190 bitbake: cooker: add providermap to dep_tree
Added providermap information to the result of buildDependTree API.

This will be used by Toaster to map virtual dependencies to recipes.

(Bitbake rev: d3e07368549f30265f59846a260efa8230a225ca)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:26:32 +00:00
Ed Bartosh 7e380d460c bitbake: taskdata: refactor get_providermap
Added optional parameter 'prefix' to filter out names that
don't start with specified prefix. Changed existing call
of get_providermap according to changed API.

Optimized the code: got rid of extra loop and temporary
list variable virts.

(Bitbake rev: df5a1392d6f91ccb44a99721c7d847da242121bb)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:26:32 +00:00
Richard Purdie 46731daa5f bitbake: main/runqueue: Add --setscene-only option to bitbake
Its turning out that we really need a way to have bitbake just run
the setscene tasks but not any real tasks, particularly for SDK
operations.

Add an option for this since its pretty straight forward. This allows
various nasty workarounds in OE-Core to be removed.

(Bitbake rev: e4a2aafa1650a227a04d92a8a0b31efaed2c310e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:26:32 +00:00
Richard Purdie 221d864042 bitbake: fetch/git: Change to use clearer ssh url syntax for broken servers
Some servers, e.g. bitbucket.org can't cope with ssh:// as part of
the git url syntax. git itself is happy enough with this but you
get server side errors when using it.

This changes the git fetcher to use the more common ssh url format
which also means we need a : before the path.

Seems a shame to have to do this due to broken servers however
it should be safe enough since this other form is the one most people
use on the commandline so it should be safe enough.

[YOCTO #8864]

(Bitbake rev: 4193e99adce8e88f12ac88d7578ad39575f7e346)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-08 13:58:08 +00:00
Richard Purdie 46d62d076d bitbake: knotty: Use non-interactive mode as fallback for dumb terminals
TERM=dumb bitbake X

shows no output for task status which is suboptimal. Use the non-interactive
mode if the terminal doesn't support what we need for interactive mode giving
a better user experience. Also print a note to the console to say this has
happened.

[YOCTO #8768]

(Bitbake rev: 6f84cf4bd77f35fcd07e0b2f5149f1d6866a414d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-08 13:58:08 +00:00
Ross Burton bfa7859ffa bitbake: cooker: fix findFilesMatchingInDir documentation
The documentation for findFilesMatchingInDir() was inconsistant with the
implementation: the regex was escaped before searching so effectively it's a
pure textual substring, and the machine example was broken.

(Bitbake rev: 6bef981488ec94b46dbe3797acfecf9c4b6ecbbc)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-08 13:58:08 +00:00
Ross Burton 3d4273716d bitbake: cooker: use in instead of count
No point counting all instances when we just want to know if there's any or not.

(Bitbake rev: 3369072efb653339da8dbd1ca864ff8e1ff899ca)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-08 13:58:08 +00:00
Richard Purdie f8185ff9f8 bitbake: ast: Add filename/lineno to mapped functions
Where we add in mappings for EXPORT_FUNCTIONS, add dummy filename
and lineno data so ensure the assumption that all python functions
have this is correct.

(Bitbake rev: 547128731e62b36d2271c4390b3fee2b16c535dc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:35 +00:00
Ed Bartosh a178c5a163 bitbake: main: kill server without queue setup
Setup of event queue includes registering of UI handler.
This operation can fail when cooker is busy. However, there is
no need in registering UI handler for terminating the server.

Moved the call of connection.terminateServer before setting up
of the event queue. This should make terminating server to work
more reliably as it doesn't depend on setting up the event queue
and registering UI handler anymore.

This should also help Toaster backend to restart bitbake server
and observer without getting "Could not register UI event handler"
errors.

[YOCTO #8776]

(Bitbake rev: 0c5a9349f797d05c282c2ada1893e187e05f0576)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:35 +00:00
Ed Bartosh 773700dbc3 bitbake: xmplrpc: split connect method
Current code in connect method sets up event queue, which requires
registering UI handler. This functionality may not be needed for
some operations, e.g. for server termination.

Moved functionality of setting up event queue in from 'connect'
method to 'setupEventQueue' in BitBakeXMLRPCServerConnection class.

(Bitbake rev: 4429871da76d6bd29e023ff42740fe7daa6b40fa)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:35 +00:00
Ed Bartosh 05b4fbc947 bitbake: uievent: refactor retry loop
Replaced 'while' loop with 'for' loop.
Made the code more compact and hopefully more understandable.

(Bitbake rev: 4e1e497c8432536b3522295e5b1284844ccea056)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:35 +00:00
Ed Bartosh ebc169c360 bitbake: uievent: get rid of EventHandler attribute
This attribute was introduced by mistake. EventHandle is used in the
code for the same purpose.

(Bitbake rev: 8d505ec8913a7d51de48b4f52bb64c5d6a0bb08e)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:35 +00:00
Ed Bartosh 4e0de6eca2 bitbake: uievent: add error to registerEventHandler return
Current code throws Exception("Could not register UI event handler")
if event handler can't be registered. The real reason of this is that
cooker is in busy state. Error message lacks information about this.

Added error message to the return value of registerEventHandler.
Included returned error message into the log message and exception
text.

(Bitbake rev: 07de1ca7d57dcd0cc37406feae2949da12a3fa7a)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:34 +00:00
Ed Bartosh 01419d5565 bitbake: cooker: add state.get_name method
Added method to convert state code into the human readable name.
It will be used in logging and error reporting.

(Bitbake rev: 9ec6379b27d210214d0b3f2e55962f721b7f5f51)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:34 +00:00