Commit Graph

4460 Commits

Author SHA1 Message Date
bavery a9dc72f08b bitbake: hob: removal of hob ui and associated ui files
We've been gearing up the Toaster web UI to replace the Hob (GTK+ based) UI
for some time now; Hob has basically been on life support for the past few
releases. As of late last month in master, Toaster has the capability to
select the packages in an image, removing the last thing that Hob could do
that Toaster couldn't.

To recap, the reasons why Hob is being removed include:

- The code is tightly woven into BitBake, making it fragile. This means it
needs significant QA and maintenance on an ongoing basis.

- Some of the implementation is not ideal; we'll be able to remove some cruft
from BitBake and OE-Core at the same time.

- It's GTK+ 2 based, not the current GTK+ 3.

- Toaster is now a much more capable UI and is being actively maintained

The discussion about removing hob can be found at:
http://lists.openembedded.org/pipermail/openembedded-architecture/2016-February/000082.html

(Bitbake rev: be2cceea159c6ca9111eff3df87b98513eab6d72)

Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-02 22:41:23 +00:00
Brendan Le Foll 27468dbc3a bitbake: fetch2/npm: Add missing ParameterError import
(Bitbake rev: b583a40c9086b3587065995d8cee45b7ea36c1b3)

Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-02 22:41:22 +00:00
Brendan Le Foll 44e3461969 bitbake: npm: in cases where shrinkwrap resolved a git URL, ignore it and grab dist.tarball
npm-shrinkwrap will sometimes resolve a git URL which instead of a http url, in
this case go and grab the dist.tarball via npm instead of using the resolved
URL.

(Bitbake rev: eb53b927ff59aa19cf28bc46beb9f9a185a59990)

Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-02 22:41:22 +00:00
Richard Purdie 2a7318133b bitbake: fetch2: Fix unpack for absolute file urls
The previous commit breaks absolute pathnames in file:// urls, this
fixes it.

(Bitbake rev: b8113a1800687a37a26ac28deafdbafd74cc138e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-02 22:41:22 +00:00
Alexander Shashkevich 865d2feff6 bitbake: fetch2: fixes copying of file://dir; subdir=foo, bug 6128 and bug 6129
When in SRC_URI appears file://dir;subdir=foo unpacker copies 'dir' to ${WORKDIR}, not
${WORKDIR}/foo as it should be.

These changes are fixing following bugs as well:
Bug 6128 - Incorrect wildcard unpack behaviour in fetcher
Bug 6129 - Local directories unpack to a different location than local files

(Bitbake rev: e659a3b0c2771679057ee3e13cd42e6c62383ff2)

Signed-off-by: Alexander Shashkevich <alex@stunpix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-02 22:41:22 +00:00
Richard Purdie 2e27c4bf23 bitbake: fetch2/npm: Enable fetcher
For some reason the enablement piece of the patch went missing, add it.

(Bitbake rev: 0270b5a3873ed0aeca3a66198c87a6164fb644b8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 12:01:07 +00:00
Richard Purdie 030d920e2d bitbake: providers: Fix PREFERRED_VERSION lookup for '_' in PN
PN can contain '_', e.g. gcc-cross-x86_64 and an override cannot
hence we do this manually rather than use OVERRIDES.

(Bitbake rev: 7a6baf02617d1edced4eaff235e73d746e2a3b68)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:34:38 +00:00
Richard Purdie c679a3dd41 bitbake: fetch2: Skip lockfiles and donestamps for local files
For local files, there are no races with downloads, we don't need ".done"
stamps and we don't need lockfiles.

This considerably cleans up DL_DIR and all the pointless ".done" files
as well as removes stalls over local files with the same name.

(Bitbake rev: 48e903745db578d9b9b425a8d411c1369df0eb94)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:34:38 +00:00
Richard Purdie d01042eda9 bitbake: fetch2/__init__.py: Error if lockfile path invalid
Rather than create ".lock" and ".done" files with no name, error,
forcing us to fix the cases where this is a problem.

(Bitbake rev: 81158071508cc68c39db7d501370872f44d335cc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:34:38 +00:00
Richard Purdie ab7b7bf94d bitbake: fetch2/__init__: Fix decodeurl to better handle urls without paths
If we specify urls such as npm://somehost;someparams the fetcher currently
does a poor job of handling mirrors of these urls due to deficiencies in the
way decodeurl works. This is because "somehost" is returned as a path, not
a host.

This tweaks the code so that unless its a file url, the host is returned
correctly.

This patch also adds test cases for these urls to the exist set of test
urls.

We need to tweak the URI() class since this thinks this is a relative url
which is clearly isn't. We also need to handle the fact that encodeurl will
error if passed a url of this form (it would want the path to be '/'.

(Bitbake rev: 83203cd2e677706e0111892a7843b83263cb8bd9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:34:38 +00:00
Richard Purdie 06b4d8ffe4 bitbake: fetch2/wget: Set localfile for directories
If we wget something which looks directory like we end up with lock files
and done stamps without names, they also all use the same lockfile.

This change ensures that we use separate lock files based on the url
and avoid creating the mysterious ${DL_DIR}/.done files.

(Bitbake rev: 20bc82086018832e047345a672d74b6c1c113650)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:34:38 +00:00
Belen Barros Pena e0fd96442a bitbake: toaster: change 'delete layer' to 'remove layer'
I have received quite a few complaints about the use of the word
'delete' for layer removal, so change it to 'remove'. That also matches
the language we use for packages in image customisation.

[YOCTO #9165]

(Bitbake rev: 3c5ac2ddfb3f5ecd3f3218de0d6564e5f3842b98)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:38 +00:00
Belen Barros Pena 6e8282063f bitbake: toaster: rename 'run again' button
I quite dislike the 'Run again' label we use in the button that rebuilds
things. Changing it to 'Rebuild', which is shorter and more specific.

(Bitbake rev: 865a2015e86a1bc5cc7d63308f27c292d1ca98eb)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:36 +00:00
Dave Lerner c8dd72c797 bitbake: toaster: fix banner after customimage package add
Correct formatting of the banner message after adding a package to a
custom image.

[YOCTO #9101]

(Bitbake rev: da233005eb8cfa7842cd1a768c16e42aaaa55fad)

Signed-off-by: Dave Lerner <dave.lerner@windriver.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:35 +00:00
Belen Barros Pena 149f57470c bitbake: toaster: custom breadcrumb for the default project
The default "Command line builds" project does not have a Configuration
page. It therefore needs a custom breadcrumb where the project name goes
to the project builds page, instead of the project configuration page.

(Bitbake rev: 5545acf6703a25ee46776138bbbd804615add89c)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:33 +00:00
Haris Okanovic 4a12865c7b bitbake: prserv: Add dump_db()
Returns a script (string) that reconstructs the state of the
entire database at the time this function is called. The script
language is defined by the backing database engine, which is a
function of server configuration.
Returns None if the database engine does not support dumping to
script or if some other error is encountered in processing.

The SQLite3 implementation in db.py calls iterdump() [1] to generate
a script. iterdump() is the library equivalent of the `sqlite3 .dump`
shell command, and the scripts are compatible. Execute the script in
an empty SQLite3 database using the sqlite3 utility to restore a backup
of prserv.

Use case: Backup a live PR server database in a non-racy way, such
that one could snapshot the entire database after a set of bitbake
builds all using a shared server. I.e. All changes made prior to
the start of a dump_db() operation should be committed and captured
in the script. Subsequent changes made during the backup process are
not guaranteed to be captured.

Testing: ~7MB database backs up in ~1s while PR server is under load
from 32 thread bitbake builds on two separate machines.

[1] https://docs.python.org/2/library/sqlite3.html#sqlite3.Connection.iterdump

(Bitbake rev: 004003daf6bd0f0233ce5c2d95f1d7d64ab91bb3)

Signed-off-by: Haris Okanovic <haris.okanovic@ni.com>
Reviewed-by: Ken Sharp <ken.sharp@ni.com>
Reviewed-by: Bill Pittman <bill.pittman@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:31 +00:00
Dave Lerner bdb51abb3c bitbake: toaster: remove custom images from Image Recipes
Fix the view of 'Image Recipes' under 'Configuration' to only show
image recipes that are not customised since custom images have their
own page.

[YOCTO #9111]

(Bitbake rev: 18a93b360301a5497d5c8ef74ab71f374f2ad210)

Signed-off-by: Dave Lerner <dave.lerner@windriver.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:29 +00:00
Elliot Smith 98d462c8e6 bitbake: toaster: show suffix for image files and basename for artifact files
The build dashboard doesn't show image and artifact files correctly,
as it shows the full filename for images and the filename plus
path relative to DEPLOY_DIR for artifacts.

Instead, show just the suffix for image files, and the basename
for artifact files.

(Bitbake rev: 8084dcdc283b4dc170f066c202f89d56ce1abbef)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:28 +00:00
Belen Barros Pena 88b56603ce bitbake: toaster: add missing link to image recipe details
In the 'New custom image' page, each image recipe name listed should
link to the corresponding image recipe details page, so that users can
look into what packages are installed by a certain image, and decide
based on that if they want to customise it or not.

This patch adds that missing link.

(Bitbake rev: a481af693bfef0171732c18c298e285986b82de3)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:26 +00:00
Belen Barros Pena 25b179d5c3 bitbake: toaster: adjust the search field width
The search field at the top of our tables was using one of the Bootstrap
classes for text field sizing. Those classes are a bit rigid, resulting
in text fields too wide that made other table controls wrap.

Setting a maximum width to the search form using one of the span classes,
combined with a % width css declaration, make for text fields that
adapt a bit better to the horizontal space available in each table.

(Bitbake rev: 7833fab2e03f2d9a01ab9ad0a13c190382098b5e)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:24 +00:00
Belen Barros Pena a97081ba55 bitbake: toaster: make 'configuration' the first tab
Our project pages have 4 tabs: builds, configuration, import layer and
new custom image. Even though we treat the 'configuration' as the
default tab, it comes second after the builds tab.

That's a bit strange: the default tab should be the first one listed.
This patch changes the tab order to put 'configuration' first.

(Bitbake rev: ccb90019489c2c324c2a5a60295e02280a2ec18f)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:23 +00:00
Belen Barros Pena e1fc3190b7 bitbake: toaster: link to configuration in all breadcrumbs
The existing breadcrumb does not always provide a link to the project
configuration page. When you are in the build history pages, you must go
back to the builds information first, and from there access the project
configuration. That feels very long.

Change the breadcrumb so that the project name item always provides a
link to the project configuration.

(Bitbake rev: 9910f3f292d35fc91215d550c5f123dcf18ab35d)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:21 +00:00
Belen Barros Pena df2808f5f6 bitbake: toaster: reduce max height of modal dialogs
Now that we use modal dialogs to display dependency information for
packages, we are hitting their maximum height relatively often. It is set
by default to 400px, which makes it a bit tight at a 1280x800 viewport
size.

Reduce the maximum height to 300px to make things a bit more
comfortable.

(Bitbake rev: e36001d61768979d66cba0f3d4f5a2aaf4af2cb7)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:18 +00:00
Belen Barros Pena 6c51f0850f bitbake: toaster: disable add layer button on click
The 'add layer' button in the project configuration page remains enabled
after you add a layer. If you click it again, the same layer you just
added is added again.

This patch disables the 'add layer' button on click, to avoid this bit
of weirdness.

[YOCTO #8905]

(Bitbake rev: 63705f60035884a810fdd36e5a3fe10e411f23c7)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:16 +00:00
Belen Barros Pena d4a663a701 bitbake: toaster: apply error class to name field
The form for naming new custom images shows you an error message when
the name already exists or you include an invalid character in it. But
when an error appears, the input field was missing the red highlight.

This patch applies the right class to the form controls whenever an
error message is shown.

(Bitbake rev: df342e7662179410467c47cd870180ea75f863d4)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:15 +00:00
Belen Barros Pena 48f0ae2c12 bitbake: toaster: fix custom image name form
The placeholder text in the form where you name your new custom images
didn't display fully. This patch fixes the styles so that the text shows
properly. It also changes the text itself to make it a bit shorter.

[YOCTO #9122]

(Bitbake rev: 9df802182b0b96295b148a1681c2265e72d8306b)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:12 +00:00
Belen Barros Pena 07eb4f274c bitbake: toaster: comment out project release change
Although the support for building more than one release and how we
handle the build directories is the subject of lively discussion, we all
seem to agree on removing the ability to change the release of a
project. The feature is currently not working but exposed to users,
which is not a happy state of affairs.

This patch comments out the controls that give access to the release
changing functionality to hide them from users, but does not touch
anything else. Once all moving pieces start to settle down, we can make
a final decision regarding this feature, and clean up the code
accordingly.

[YOCTO #8917]

(Bitbake rev: 3a8c6f7155517cd61a160595b81e7bed84ba4eaf)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:11 +00:00
Richard Purdie 12ade9b093 bitbake: fetch2/npm: Add mirroring support for npm fetcher
(Bitbake rev: bff46c614d3f9cc18a1c8908c47842712e0e3a8c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:09 +00:00
Brendan Le Foll ca5b6d66e0 bitbake: fetch2/npm: Add npm fetcher
npm fetcher with support for shrinkwrap files and lockdown files to easily
download and install an npm package with strict dependency resolution.

The SRC_URI should be in the format of:
SRC_URI = "npm://registry.npmjs.org/;name=${PN};version=${PV}"

To add a shrinkwrap and lockdown file use:
NPM_SHRINKWRAP := "${THISDIR}/${PN}/npm-shrinkwrap.json"
NPM_LOCKDOWN := "${THISDIR}/${PN}/lockdown.json"

(Bitbake rev: dec75bbc5d075acb322dad8b1c40d6bd518dc9fd)

Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:07 +00:00
Brendan Le Foll 813bd1f806 bitbake: utils.py: Add sha1_file call
This is useful as npm-lockdown uses sha1 because npm releases the sha1 of
packages and whilst this is undocumented it seems no other algorithm is
supported

(Bitbake rev: fd5d9011f6dd7029895b64d8a02d33185b9aa8ae)

Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:05 +00:00
Michael Wood 8c0ba8d9af bitbake: toaster: toastergui Fix invalid char test and implementation
This test was passing but because the assertion was the wrong way round
and should have expected the first one to pass and second one to fail,
in reality both were failing as the method for checking the invalid char
was incorrect.

(Bitbake rev: 932a92b8130d4815656dc885f0c6e4afa4502022)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-19 15:38:49 +00:00
Michael Wood 913e9b1cbc bitbake: toaster: PackagesTable show only installed packages
When showing the package list for the image recipe details only show the
packages which are installed in the image rather than all the packages
which are produced.

[YOCTO #9108]

(Bitbake rev: be7dca6c7607c0d13151c2d3f7ad7adcdf365076)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-19 15:38:48 +00:00
Michael Wood 94bca58067 bitbake: toaster: toastergui unit tests convert to use fixtures
Instead of having to manually write up in the setUp function all the
different objects to create in the database, just specifiy them in a
fixture and load it. This has the advantage that it can be used on a
live server by using the ./manage.py loaddata command and can then
be inspected.

(Bitbake rev: 8a653ad8b075892d24d15696b3070c5a28a4fbe2)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-19 15:38:48 +00:00
Michael Wood 8796ac8a55 bitbake: toaster: SoftwareRecipesTable apply default order_by
Make sure the default orderby for the SoftwareRecipesTable is applied

(Bitbake rev: 1688608b537d8de840c6d1e4802ae41ca872e5bf)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-19 15:38:48 +00:00
Michael Wood 8469e58025 bitbake: toaster: orm migrations Sort out migrations mess
We messed up the migrations by squashing some of the image
customisation model definitions into the initial migration which
has meant some irreversible operations on mysql took place.
This deletes, re-orders and fixes the migrations.

If your schema is up to date you may want to use ./manage migrate
with --fake or --fake-initial to avoid re-applying migrations.

[YOCTO #9116]

(Bitbake rev: 19bd63fc3a28dcbd0f531a5b06a037da34568bac)

Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-19 15:38:48 +00:00
Richard Purdie 4e5e501155 bitbake: data_smart: Drop default expand=False to getVarFlag [API change]
At some point in the future, getVarFlag should expand by default. To
get there from the current position, we need a period of time where the
expand parameter is mandatory.

This patch starts that process. Clear errors will result from any code
which doesn't provide this. Layers can be fixed with an expression
like:

sed -e 's:\(\.getVarFlag([^,()]*, [^,()]*\)):\1, False):g' -i `grep -ril getVar *`

(Bitbake rev: aa3faebdf6af66ab34f74d328b2113de0b08c7ee)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:48:00 +00:00
Richard Purdie c7610aa353 bitbake: data_smart: Drop default expand=False to getVar [API change]
At some point in the future, getVar should expand by default. To get
there from the current position, we need a period of time where the
expand parameter is mandatory.

This patch starts that process. Clear errors will result from any code
which doesn't provide this. Layers can be fixed with an expression
like:

sed -e 's:\(\.getVar([^,()]*\)):\1, False):g' -i `grep -ril getVar *`

(Bitbake rev: fab717d303df0bcef737661f6917f275f35215a4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:48:00 +00:00
Markus Lehtonen 4f0ab27c6a bitbake: SignatureGeneratorBasic: make checksum cache file configurable
Define a new bitbake configuration variable BB_HASH_CHECKSUM_CACHE_FILE
that can be used to define the cache file to use for file checksum
cache.

(Bitbake rev: a965b390d6240e279c190b92b17c0573e9bd604c)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:41:16 +00:00
Markus Lehtonen 0cdf1934fd bitbake: MultiProcessCache: make cache filename configurable
If no cache file name is given a default from class variable is used,
like before.

(Bitbake rev: 2602a312818f564961de7dfa63c429d45ff9e5ac)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:41:16 +00:00
Markus Lehtonen ca552bb4b1 bitbake: FileChecksumCache: add get_checksums() method
Move the local file checksum functionality from bb.fetch2 into
bb.checksum module.

(Bitbake rev: 4f60933283f377d68f191db849dac6c1dc7a0aed)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:41:16 +00:00
Markus Lehtonen 8f61f2d881 bitbake: bb/runqueue: save task file dependency cache onto disk
Before this patch the usage of cache was quite useless as the file
checksums were not actually cached on disk but re-calculated every time.
This patch utilises the new writeout_file_checksum_cache() method of the
SignatureGenerator class to do the job.

(Bitbake rev: 5ac9cbf405841ed3f65e6f99a3cee032567fb182)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:41:16 +00:00
Markus Lehtonen 5177b1e4ac bitbake: SignatureGenerator: add method for saving the file checksum cache
Extend the API in order to be able to write out the file checksum cache
onto disk. SignatureGeneratorBasic class now implements a method that
update the fetcher local files checksum cache with the task file
dependency checksums.

(Bitbake rev: ecdabd321d48fa367b89ebffc00aa525b6eaa95c)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:41:15 +00:00
Markus Lehtonen 97617fd675 bitbake: bb/cache: drop some unused arguments
Drop unused 'd' argument from the cache save methods, simplifying the
API.

(Bitbake rev: 81bc1f20662c39ee8db1da45b1e8c7eb64abacf3)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:41:15 +00:00
Mirela Rabulea 5a87d8c477 bitbake: Allow Hob to run images on a custom simulator, other than qemu
The current behavior of Hob is that there is a "Run Image" button which becomes visible only for qemu images.

My suggested change is:
- if an image is selected and it is qemu-compatible, let the "Run image" button be named "Run qemu image"
- if an image is selected and it is not qemu-compatible, let the same button show up with the name "Run custom image", and besides that, an option shows-up to allow the selection of the custom script (by default it points out to runqemu script) to be used for launching this custom image

Note: in case there is more than one toggled image (qemu runnable or deployable), when the user clicks the "Run custom image" button, a dialog will be presented, allowing to choose between any of the existing images.

[YOCTO #8940]

(Bitbake rev: cc4cfc2370297b8feb2dc39d4262e73adf06c09a)

Signed-off-by: Mirela Rabulea <mirela.rabulea@nxp.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:41:15 +00:00
Christopher Larson a25858982a bitbake: bb.ui.knotty: prefix task messages with recipe/task
An example prefix: `perl-5.22.1-r0 do_compile:`

(Bitbake rev: 792b759e59e31d2e43d525a6e50d866b4f51f072)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-16 09:04:23 +00:00
Christopher Larson 4bf8b21ab8 bitbake: Move bb.{debug,note,..} into their own logging domain
This lets us filter and use -l to show messages from that source specifically.

(Bitbake rev: 7946927156dec33364418988eb921ddb273660eb)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-16 09:04:23 +00:00
Richard Purdie 8b217208cf bitbake: knotty: Set exit failure code on runQueueTaskFailed events
If the worker segfaults, we may never see a TaskFailed event from it, only
a runQueueTaskFailed event. In this case, return_value isn't getting set
leading to an incorrect exit code from bitbake. Fix by setting return_value
in both places.

(Bitbake rev: e5dd50e0d95d532fe31dde61f8c6b1a7a72321e9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-15 16:48:12 +00:00
Richard Purdie a9223e20d1 bitbake: taskdata: Fix traceback issue with missing provider
If there is a missing provider and we're using "-k" mode alongside "-w",
we could get a traceback since there was no provider. Add tests to avoid this.

(Bitbake rev: 90a4805e4e770a433b4394ea99792731e9a4b546)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-15 16:48:12 +00:00
Richard Purdie 7593756cbb bitbake: cooker: Improve cache handling
We're seeing issues where the self test, which uses tinfoil doesn't
notice the changed contents of include files. The issue is
cached_statements in the parser being reused when the files have changed.

Whilst looking at this, I realised there were some other issues:

* We need to also invalidate the mtime cache when cooker restarts
* We should pass full filenames to the file invalidation code
* We should process cached_statements as part of inotify invalidation

With these fixes, the caching is more reliable for memory resident
bitbake too. It does raise some questions about cache validation and
lifecycles and indicates bitbake does need more work in the area,
preferably with the removal of the globals. This at least highlights
and works around some of the current issues.

(Bitbake rev: 3f507ff8bc467fba936cf3f31bb8ea8e02f168e8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-15 16:48:12 +00:00
Richard Purdie 1c2d632a02 bitbake: Bump version post release to 1.29.0
(Bitbake rev: da1107a2c2331b3335c7bd46821edcb1933a0ff8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-12 13:44:22 +00:00
Ed Bartosh f9f93ae856 bitbake: cooker: gracefully shutdown parsers
CookerParser.shutdown code doesn't do all required work to shutdown
parser processes. As a result bitbake hangs if interrupted during
parsing. Putting None into the parser_quit queue should fix this issue
as it makes parsers to quit main loop.

(Bitbake rev: f67307977e8f089ce6d208d3e9de2a6a1768757e)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11 22:34:19 +00:00
Elliot Smith 1f7f077466 bitbake: buildinfohelper: unset brbe variable when build finishes
The brbe variable is set on the bitbake server when the Toaster
UI starts. This enables Toaster to associate events with the
correct build and build environment.

However, the brbe variable is also used when a build starts to
identify whether a new build needs to be created, or an existing
one looked up. This causes a bug for command-line builds which
happen after a Toaster-triggered build: because the brbe variable
is never unset on the server or the buildinfohelper, the new
command-line build events are treated as originating from the
previous build.

Ensure the brbe variable is reset when the buildinfohelper "closes"
a build, so that each build then either sets the brbe variable
(Toaster-triggered builds) or leaves it blank (command-line builds).

Also modify the localhostbecontroller so that the brbe variable
is not set on the server and not looked up from the server. This
ensures that it is only set when the triggerBuild() method is
called, and that it remains as None for command-line builds.

[YOCTO #9021]

(Bitbake rev: 4a6a8d0074f62208d843b06344be31ae73d9b745)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11 22:34:19 +00:00
Mihail Stanciu 19045bab71 toaster: tests Remove symlinks from toasteruitest folder
Remove symlinks in the UI tests folder as they are causing
problems for bitbake upstream.

[YOCTO #8787]

Signed-off-by: Mihail Stanciu <stanciux.mihail@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11 17:07:39 +00:00
Richard Purdie 400f53003e bitbake: build: Improve python execution tracebacks
If something fails in a exec_func_python() the current stack trace shows
incorrect filenames and linenumbers. For example:

The stack trace of python calls that resulted in this exception/failure was:
File: '/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb', lineno: 200, function: <module>
     0196:                chksum = bb.utils.sha256_file(fn)
     0197:                f.write('%s\t%s\n' % (chksum, os.path.relpath(fn, baseoutpath)))
     0198:
     0199:
 *** 0200:copy_buildsystem(d)
     0201:
File: '/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb', lineno: 9, function: copy_buildsystem
     0005:IMAGE_FEATURES += "splash package-management x11-base x11-sato ssh-server-dropbear hwcodecs"
     0006:
     0007:LICENSE = "MIT"
     0008:
 *** 0009:inherit core-image
     0010:
     0011:IMAGE_INSTALL += "packagegroup-core-x11-sato-games"
File: '/usr/lib/python2.7/subprocess.py', lineno: 535, function: check_call
     0531:    The arguments are the same as for the Popen constructor.  Example:
     0532:
     0533:    check_call(["ls", "-l"])
     0534:    """
 *** 0535:    retcode = call(*popenargs, **kwargs)
     0536:    if retcode:
     0537:        cmd = kwargs.get("args")
     0538:        if cmd is None:
     0539:            cmd = popenargs[0]

The problem is the use of "FILE" to obtain the current filename. Instead,
we therefore inject the function being executed into the methodpool which
allows us to correct its linenumber and filename information. We can then
clearly mark the initial piece as autogenerated and the rest of the linenumber
and filename information should be correct. Afterwards the trace starts:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:copy_buildsystem(d)
     0003:
File: '/media/build1/poky/meta/classes/populate_sdk_ext.bbclass', lineno: 66, function: copy_buildsystem
     0062:    import glob
     0063:    import oe.copy_buildsystem
     0064:    import subprocess
     0065:
 *** 0066:    subprocess.check_call("foo")
     0067:
     0068:    oe_init_env_script = d.getVar('OE_INIT_ENV_SCRIPT', True)
     0069:
     0070:    conf_bbpath = ''
File: '/usr/lib/python2.7/subprocess.py', lineno: 535, function: check_call
     0531:    The arguments are the same as for the Popen constructor.  Example:
     0532:
     0533:    check_call(["ls", "-l"])
     0534:    """
 *** 0535:    retcode = call(*popenargs, **kwargs)
     0536:    if retcode:
     0537:        cmd = kwargs.get("args")
     0538:        if cmd is None:
     0539:            cmd = popenargs[0]

We can't inject into methodpool at parsing time, since there may be
_append or other override operations against the function before its
execution.

(Bitbake rev: fae153095d23157dd7e72c29f683f86149ee33a8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 15:47:11 +00:00
Richard Purdie aece748762 bitbake: build/data: Don't expand python functions before execution [API change]
Right now, if you have some python code like:

X = "a"

def somefunction(d):
    d.setVar("X", "b")
    d.setVar("Y", "${X}")

then any sane person would expect that Y = "b" at the end of the
function. This is not the case, Y = "a".

This is due to the python function being expanded before execution, the
executed code would read d.setVar("Y", "a"). This understandably
confuses people, it also makes it near impossible to write ${} in a
python function without unintended things happening.

I think there is general agreement we should fix this and standardise
on non-expansion of python functions. We already don't expand anonymous
python (mostly).

I've checked OE-Core with buildhistory before and after this change and
there were a small number of issues this exposed which I've sent
patches for.

I propose we default to not expanding python code and then deal with
any consequences from that if/as/where identified. This will improve
new user understanding and usability of the system, it also allows
several long standing weird expansion issues to be fixed.

(Bitbake rev: 8bf33a8e92c0e188fa392030025756196c96fcbb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 15:47:11 +00:00
Richard Purdie e39cfb1f9c bitbake: cooker: Don't expand python functions in variable dumps
We don't want to expand python functions since they aren't expanded
at execution time (e.g. anonymous python). They can also have side
effects.

This function is primarily used by toaster for variable dumps for later
display. The lack of expansion of python functions won't matter in this case
and actively helps some variable handling (e.g. SRCPV).

(Bitbake rev: 3f5520b4844a4bdd615046479ba08ed192bdc8cd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 15:47:11 +00:00
Richard Purdie f652b6b9d0 bitbake: data: Don't expand python functions for variable dependencies
Expanding python functions for variable dependencies doesn't really make sense,
not least since this causes execution of any inline python, it also makes it
impossible to write expressions like d.expand("${X}") of d.setVar("X", "${Y}")
which may have the wrong values if expanded now.

This starts to standardise the approach across bitbake for handling python code.

(Bitbake rev: 765a2480dbe288f64562a9611dd93b6b6dd0a64e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 15:47:11 +00:00
Richard Purdie d3e0c448ea bitbake: data_smart: Avoid expanding anonymous python functions
We don't expand anonymous python before execution, so nor should
we do this when calculating checksums for them.

(Bitbake rev: 5f10987edda35b08970a6dd6ccf9febad271ce3e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 15:47:11 +00:00
Michael Wood e0eb2ea488 bitbake: toaster: models Remove manual transaction control from lsupdates
Revert "toaster: models.py do not use transactions".
This reverts commit e248a503ba
(Bitbake rev: 48d0d510816346073f135bb86fb7904fdb50bb27)

Manually managing the database transactions caused more
problems than it temporarily solved so we return control back to
Django's orm.

[YOCTO #8796]

(Bitbake rev: 25c531915b6f8f79a0822996ceb97f90483e359f)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:33:40 +00:00
Belen Barros Pena 48622e1d6b bitbake: toaster: build section Improve display of builds when > 1 targets
* Display always the first target in alphabetical order to match what we
do in the breadcrumbs and the build dashboard heading

* Remove the extra space between the '+' and the additional number of
targets

* Make sure the tooltip with the full target list takes the Bootstrap
tooltip styles

* Replace the word 'targets' in the tooltip with 'recipes', since that's
how we call build targets everywhere else in Toaster

(Bitbake rev: 3b8747d0af4b9164e973940ed97751c951e74110)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:33:40 +00:00
Belen Barros Pena 4d0ba0fca8 bitbake: toaster: templates make build data breadcrumb consistent
The pages in the build data section of Toaster showed different
breadcrumbs: in some pages the machine was displayed, but not in others.
For builds with more than one target, some pages showed the first
alphabetical target (the correct behaviour), others didn't.

This patch removes the inconsistencies, showing exactly the same
breacrumb across all pages in the section.

The patch also removes the extra space between the '+' and the number of
targets when the builds have more than one target.

Remove an unneeded debug message

(Bitbake rev: 9cdbb543311b6f4a8a88c27fc157d998242444ee)

Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:33:40 +00:00
Richard Purdie 99184d7879 bitbake: BBHandler/ast: Merge handMethod and handleMethodFlags
The functionality overlap between these two functions is significant and
its clearer to handle both things together since they are intimately
linked. There should be no behaviour change, just clearer code.

(Bitbake rev: 391aa4afc91be90d8d3ee47e1bf797d6ebe61a71)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:32:46 +00:00
Richard Purdie 6ba69b4383 bitbake: utils: Drop datastore function inspection during exception
When we use functions from the data store, they now have correct line number
and filename information. This function would attempt to correct line numbers
which doesn't need correcting, leading to misleading messages to the user.
Therefore remove this code as being obsoleted.

(Bitbake rev: 918bec86bc8ee94feb82380ff410d9fdcbe9e720)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:32:46 +00:00
Schroeder, Henning f8a44b160d bitbake: cooker: extended dot styling
Extended the dot styling of dependencies created by bitbake -g in dot syntax to differentiate between the various kinds.
  depends: solid
  rdepends: dashed
  rrecommends: dotted

The change observed is that depends get an explicit style which is the same as dot default behavior and the runtime recommends get
dotted while before they were dashed. This helps to distinguish them graphically as well as eases post processing by script.

(Bitbake rev: 86e78e0ca7aa5452411f35239942ecee3d8824ec)

Signed-off-by: Henning Schroeder <henning.schroeder@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:32:46 +00:00
Michael Wood 30c132bc47 bitbake: toaster: Enable Image Customisation feature
Remove environment variable to toggle Image customisation feature

(Bitbake rev: 2e9c86229b8f924a5b62987f4b166f63392f12e8)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:22 +00:00
Michael Wood 5e14a8f2fe bitbake: toaster: xhr_customrecipe_packages Add dependencies to included packages
Add the dependencies of packages which are added to the
CustomImageRecipe. Currently just handle the first tier of dependencies
as this is what we show in the UI.

(Bitbake rev: 5c44609a9bf9fb23241b7dd7c58b08901d75008d)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:22 +00:00
Michael Wood 749f5a6f1f bitbake: toaster: orm generate_recipe_content only exclude locale packages
Allow package groups in our custom image recipe. Excluding them creates
more undefined behaviour than including them at this stage. Also update
to use convenience method for returning all packages.

(Bitbake rev: 8c2e8a13badacb816c4b1178b6661600008b38af)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:22 +00:00
Michael Wood 626941104f bitbake: toaster: customrecipe page Add last successful build link and conditionals
Add link to the last successful build if there is one and add
conditionals for the ancillary recipe metadata.

(Bitbake rev: 4660aaf1c6775270f8f3d0afbb7fa2ee7a2a1563)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:22 +00:00
Michael Wood 8d5b61e495 bitbake: toaster: models Add update_package_list for CustomImageRecipe
Add a method to update the packages included list from the last
build, this effectively "synchronises" the package list from what
we think will happen at the Customise image stage with what
actually was produced with a build. It's not ideal to have this
function here but we also need to make sure that no race condition
of the user accessing this list and it being updated occurs.

(Bitbake rev: 8cf6e67a955574b33856a082bdadf3194f2b6ba4)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:22 +00:00
Michael Wood 86db0bd7f2 bitbake: toaster: orm Add last_updated field to CustomImageRecipe
Field to keep track of when the package list for the CustomImageRecipe
was last updated from a build.

(Bitbake rev: 4bd4e49f13a7625997a43f3b2e67ed42c3c8e08b)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:21 +00:00
Michael Wood 18d8b17052 bitbake: toaster: models add get_last_successful_built_target method
Add a convenience method to get the last successful build
target for a CustomImageRecipe.

(Bitbake rev: 4dde3d830cd38bbe306d83629dcb80da5fc9b027)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:21 +00:00
Michael Wood 8885b7bfd3 bitbake: toaster: pkg_dependencies_popover just show direct dependencies
In the dependencies popover just show direct dependency in the list
rather than recommends, conflicts etc

(Bitbake rev: e69b00532b011327bc2495a6fb52cfe98f0f897d)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:21 +00:00
Michael Wood 40f6effec4 bitbake: toaster: models add all_depends method for Package_DependencyManager
This convenience method returns just the Package_Dependency for
the package which are regular dependencies i.e. not RECOMMENDS or
any other types.
(Bitbake rev: bd76c22fe2aa06690b4ee25de69219ac0bf6b4d6)

Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:21 +00:00
Michael Wood a8ab1c6b25 bitbake: toaster: buildinfohelper CustomImagePackage update dependency info
Instead of keeping the original dependency information for the pool of
CustomImagePackage reset it with each new build.

(Bitbake rev: a0b97ffc7a468bad081ce3276c74728bf6830250)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:21 +00:00
Michael Wood 0fee829932 bitbake: toaster: newcustomimage_modal add frontend name validation
Add front end handling of validation response from create new
CustomImageRecipe api.

(Bitbake rev: eff66b502df8e001cd0abc25bcbd742687169619)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:21 +00:00
Michael Wood cb6d290d0b bitbake: toaster: API CustomImageRecipe check the recipe name supplied is valid
Check that the name for a new CustomImageRecipe doesn't already exist in
the project or in the database of existing recipes (e.g. from the layer
index). Also restrict the characters entered for the recipe naming
convention.

(Bitbake rev: f290d428460a07e73050ff613bc222cc8c04f5ec)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:21 +00:00
Michael Wood 5634a251ce bitbake: toaster: views CustomRecipe API add size information to the package lists
Add the file size of the packages and the total to the JSON response.

(Bitbake rev: bbbd304c49b0940a695d15273934edff95d70836)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:21 +00:00
Michael Wood 6fbceb0f12 bitbake: toaster: models Invalidate ToasterTables cache when a m2m field changes
Whem a m2m field changes we need to clear the ToasterTables cache as
this can affect the state of items in ToasterTables.
For example the CustomImagePackages being added or removed from a custom
image recipe.

(Bitbake rev: c9d7b68ee0186a71e8e75a5d87122a0328001515)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:21 +00:00
Michael Wood 998f9af193 bitbake: toaster: customrecipe Add dependency tracking to package selection
Update the states of the packages in the package selection UI to reflect
whether it's likely that 1st level dependencies for the package will be
also added.

(Bitbake rev: 119569d83c3fb1d1bd162624819b3f9c63a791c4)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:21 +00:00
Michael Wood 9976e4f169 bitbake: toaster: tables move template logic into the pkg_add_rm_btn
Instead of defining this as a string it's sufficiently large enough to
warrant its own file.

(Bitbake rev: 6b39423fe5a3ed30289a8b303329a5725f7d273b)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:20 +00:00
Michael Wood d77c247230 bitbake: toaster: CustomImageRecipe generate overwrite IMAGE_FEATURES
If we're breaking up an image recipe's packages we will also need to
override IMAGE_FEATURES to make sure the customisation is not altered

(Bitbake rev: 9fd7b05dc0cf9240f7c8e3dc77b009064fd2b0cb)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:20 +00:00
Elliot Smith 481dc11547 bitbake: toaster: make locale packages uneditable in custom image page
When the packages associated with a custom image recipe are
shown in the customrecipe editing page, locale packages are
shown in the same way as all other packages. This gives the false
impression that these packages can be removed, when in fact
they are automatically added due to the IMAGE_LINGUAS build
variable.

Modify the customrecipe page so that locale packages cannot be
removed, and provide some help text explaining why.

[YOCTO #8927]

(Bitbake rev: b2208e53c00a67a7d0345e7378e6806b8ae40fb4)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:20 +00:00
Elliot Smith a757d397db bitbake: toaster: include locale and packagegroup packages in custom image
The custom image editing page doesn't show locale and packagegroup
packages: they are filtered out of the queryset used to populate
the ToasterTable.

Rather than filtering these packages out, include them in the list
of packages which are shown.

(Bitbake rev: 38a753e7b2e9ede326856b830b25e13bdd6d0d9b)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:20 +00:00
Elliot Smith baac4589ae bitbake: toaster: update custom image package table filters
The custom image package selection filters were using the old
ToasterTable filter approach, which caused the table filter to fail.

Amend the table to use the new ToasterTable filtering API to fix
this.

(Bitbake rev: 72a4cb30842fd053e46dc56df222729cbe735162)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:20 +00:00
Elliot Smith efbffe3c29 bitbake: toaster: move recent builds query to model
The progress updater for the recent builds section makes a JSON
call to the project view URL to get progress for each build.
However, conversion of the builds pages to ToasterTable broke this,
as the JSON response no longer contained the data necessary to
populate the progress bars.

Move the recent builds query to the Build model, so that it is
accessible to the ToasterTables using it ("project builds" and
"all builds"), as well as to the "project" view.

Modify the code in the recent builds template to use the slightly
different objects returned by the recent builds query on Build.

(Bitbake rev: 5189252635ddc7b90c9a43aaed9f196c31e1dcad)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:20 +00:00
Elliot Smith b51478582f bitbake: toaster: update customimagerecipe migration
When applying migrations, Django shows this warning:

"Your models have changes that are not yet reflected in a migration,
and so won't be applied."

This is because the customimagerecipe model has changed, but those
changes are not covered by a migration.

Add the missing migration to clear this warning.

(Bitbake rev: df8185fcbd84061976d91b03b2a9268b319a6184)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:20 +00:00
Elliot Smith df58f5b70b bitbake: toaster: add merge migration to resolve conflict
Django detects a conflict between a migration added to support
image customisation and another migration which supports PROVIDES.

Add a merge migration to resolve the conflict (as suggested by
Django).

(Bitbake rev: a26bfd9d2490dc0fd90bf6d1690e63ac26001559)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:20 +00:00
Michael Wood 38f4913270 bitbake: toaster: orm generate_recipe_file_contents Handler for require recipe
Add a special case for when the recipe we have based a custom image
recipe on requires another recipe.
In this case we need to adjust the file location to be able to
require
the recipe when we're in the toaster-custom-images layer.

For example:
"require core-image-minimal.bb" is changed to:
"require recipes-core/images/core-image-minimal.bb"

(Bitbake rev: 26025e1ea49b3ebfcfd508d1608fa8c9e722ad03)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:20 +00:00
Michael Wood 769017e477 bitbake: toaster: project builds Poll the server to get latest progress for build
Poll the server for the project build progress value. This is something
that will need to be re-done once we have a proper API for this on the
server side.

[YOCTO 8328]

(Bitbake rev: ec467e43c39eadf02412b89db10c09ed78a5a9f5)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:20 +00:00
Michael Wood 971d65c614 bitbake: toaster: localhostbectrl Update the dirpath of customrecipe's base layer
We need to know the location of the based_on recipe's layer on the
file system before we try and generate the custom image recipe. As
we read the recipe to make the custom version.

(Bitbake rev: e6a7cacbddd1df5bac0b79384199cf7264c5bbd5)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:19 +00:00
Michael Wood 6d9f342e12 bitbake: toaster: tables Check layer presence in project for customise_btn
Make sure we send the current list of layers to the customise button to
be able to know whether it should be set as an add layer button or a
customise button on the New custom image page.

(Bitbake rev: 5ddb35c98b609d85f97d482b54cabe3a2812afe6)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:19 +00:00
Michael Wood 76c0008014 bitbake: toaster: toastergui tests Add addtional data to the setUp for new tables
Add additional data to the setUp to be able to test all the tables for
Image Customisation. Also add the name of the table being tested to the
num of rows assertion.

(Bitbake rev: dfcbcf789cf3f0733ca26b0601fdf97ce4291674)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:19 +00:00
Michael Wood 70a078ee85 bitbake: toaster: tables SelectPackagesTable rename recipe_id to custrecipeid
Rename the recipe_id to custrecipeid to avoid confusion about which type
of object we're going to be accessing. This means that in the unit tests
for tables we can pass a different kwargs for custom recipes vs normal
recipes.

(Bitbake rev: ae3301a1047b3efb4b340b50a10d5d585b7333da)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:19 +00:00
Michael Wood 7e4c231ca2 bitbake: toaster: toastergui tests Update package test to use CustomImagePackage
Update test for adding and removing a package from a CustomImageRecipe
so that it uses the CustomImagePackage and correct fields for the packages
included. Change the test for error condition to use an invalid package
id as ManyToMany remove() on package that isn't in the relationship does
not throw an error.

(Bitbake rev: daccb2978f833a9e7af270160331da3e9a158219)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:19 +00:00
Michael Wood 4b3c9d61dc bitbake: toaster: customrecipe Add further front end features using new API
This adds some basic package dependency hint modals when you add and
remove a package. It also makes sure that if the CustomImageRecipe has
no current included packages that we go and check this with the server
to see if a relevant build has taken place which will provide this
information.

[YOCTO #8082]

(Bitbake rev: 418f5509e74d46d36a8eb966a245083006e5f4ba)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:19 +00:00
Michael Wood b213907afe bitbake: toaster: xhr_customrecipe_packages add GET info for package response
Add response for GET to the xhr_customrecipe_packages ReST API

/xhr_customrecipe/<recipe_id>/packages/<package_id>
Thie response includes the id, name, version and dependency information
for the package.

(Bitbake rev: c45791fc85d26c43b0a3b0a459111d2ff5583540)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:19 +00:00
Michael Wood a9668eeb74 bitbake: toaster: xhr_customrecipe_id change to use CustomImagePackage
Instead of doing a shallow copy of the package into the
CustomImageRecipe when we add packages we can now use the
CustomImagePackage as a M2M field on the Package to CustomImageRecipe.
Also switch to using Target_Installed_Package as the method to retrieve
the package list from the build.

(Bitbake rev: 4ebc81823b3aec6ecf38835acad5263a81eb41c5)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:19 +00:00
Michael Wood 439314c1b2 bitbake: toaster: API allow CustomImageRecipe to be updated after creation
When we create a CustomImageRecipe we create a Layer_Version and
Recipe for that Recipe to be in, we only need one Layer_Version for our
Recipes so if that Layer_Version is updated by building it we need
a slightly more custom version of get_or_create to take into account the
fields which we expect can change but still mean that the object we want is
valid and doesn't need to be created.

In the Recipe case this is when we're updating an existing
CustomImageRecipe as we allow people to create a recipe even when the
based on recipe hasn't been built so we need to update it once a build
has happened.

(Bitbake rev: 0fe2c72ab82c6de2825a390fbb460b892a7a9cfc)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:19 +00:00
Michael Wood 9ea4de6d80 bitbake: toaster: tables Change SelectPackagesTable to use ProjectPackage
This changes the SelectPackagesTable to use the ProjectPackage table
instead of very large expensive queries to retrieve a list of currently
available packages for the project.

(Bitbake rev: 4b4b7e28d602ac5283659f806d695cc0451d292e)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:18 +00:00
Michael Wood 20f400b7bd bitbake: toaster: tables add recipe download link to CustomImagesTable
Add the download recipe link and fix the package count field.

(Bitbake rev: 85891e5320014f363dba093ac2db681d55375ee3)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:18 +00:00