Commit Graph

3154 Commits

Author SHA1 Message Date
Markus Lehtonen 292c3e82e3 bitbake: checksum: In FileChecksumCache don't follow directory symlinks
Before this patch, directory symlinks mathcing filename pattern (either
a file name or a glob pattern) were followed. However, directory
symlinks deeper in the search chain were omitted by os.walk(). Now
directory traversal behaves consistently, ignoring syminks on all
levels.

One reason for choosing not to "walk into" directory symlinks is that
dir symlinks in externalsrc.bbclass in oe-core are causing problems in
source tree checksumming.

[YOCTO #8853]

(Bitbake rev: 66dff37ebcd1dd14ebd6933d727df9cf0a641866)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-30 12:01:29 +01:00
Richard Purdie ddbeb56f4f bitbake: cookerdata: Improve handling of ParseError
If local.conf contains an invalid line, e.g.:

 APPEND += " igor"

(note the leading space) then nasty tracebacks are shown which confuse the
user. Change so the parse error is simply shown without a traceback, improving
the user experience.

[YOCTO #9332]

(Bitbake rev: 148aa1fb45dcb37a756a08301a7daf270e753180)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-29 23:21:52 +01:00
Aníbal Limón 60656d07ea bitbake: fetch2/wget.py: _check_latest_version_by_dir fix prefix detection
When prefix is part of the version directory it need to ensure that
only version directory is used so remove previous directories if exists.

Example: pfx = '/dir1/dir2/v' and version = '2.5' the expected result
is 'v2.5' instead of '/dir1/dir2/v2.5'.

[YOCTO #8778]

(Bitbake rev: c760531c6dbf88135ab9f8e6f0784ccbf2cce1e4)

Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-28 15:55:51 +01:00
Aníbal Limón 45ee2b1079 bitbake: fetch2/wget.py: _check_latest_version_by_dir use group names
Little improvement for reference tokens by names instead of index.

(Bitbake rev: e8ea15eeb1857ed4bb6337836bd2fb1f5dbb1bdf)

Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-28 15:55:51 +01:00
Richard Purdie 84d5924012 bitbake: fetch2: Handle lockfiles for file:// urls redirected to mirrors
We recently dropped lockfiles for file:// urls which in itself makes
sense.

If a file url redirects to something like an http:// mirror, we'd have
no lock taken for the original file and could race against others
trying to download the file. We therefore need to ensure there is a
lock taken in the mirror handling code.

This adds code to take such a lock, assuming it isn't the same lock
as the parent url.

(Bitbake rev: 913b6ce22cd50eac96e8937c5ffc704bfce2c023)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-26 07:34:58 +00:00
Derek Straka fda94f4e2d bitbake: bitbake: fetch2/gitsm: Fix fetch when the repository contains nested submodules
This fixes a problem when the repository contains multiple levels of submodules via a resursive submodule init.

(Bitbake rev: dbafbe229360ffe5908b106a9c10e274712b9b17)

Signed-off-by: Derek Straka <derek@asterius.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-26 07:34:58 +00:00
Ed Bartosh b58e5b1fb3 bitbake: bitbake: xmlrpc: set single use mode differently
Currently xmlrpc server implicitly sets itself into single use mode
when bitbake server is started with anonymous port (0) or no port is
provided in command line. In this mode bitbake shuts down xmlrpc server
after build is done. This assumption is incorrect in some cases.
For example Toaster uses bitbake in this mode and expects xmlrpc server
to stay in memory.

Till recent changes single use mode was always unset due to the bug.
When the bug was fixed it broke toaster builds as Toaster couldn't
communicate with bitbake server in single use mode.

Reimplemented logic of setting single use mode. The mode is explicity
set when --server-only command line parameter is not provided to bitbake.
It doesn't depend on the port number anymore.

[YOCTO #9275]
[YOCTO #9240]
[YOCTO #9252]

(Bitbake rev: afc0dd5c532684f6201b1e12bbf4c226ea19062d)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-24 12:31:50 +00:00
Ed Bartosh 2918b50361 bitbake: toasterui: remove ParseStarted from the event list
bb.event.ParseStarted event is not processed by toasterui, but
present in event list. This causes the following error:
   WARNING: Unknown event: <bb.event.ParseStarted object at ...
and non-zero return code:
   WARNING: Return value is 1

(Bitbake rev: 1cc102f3d83d9467a3a3c422254333796ba95605)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-23 17:56:16 +00:00
Michael Wood ab2abd46f6 bitbake: toasterui: Remove the excessive exception logging
Remove the very verbose log dump from toasterui. This generates several
megabytes of not that useful debug information and actually hinders
finding the original exception.

(Bitbake rev: a21dc134bdce2c9eb5e47c770094660f0c45c398)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-23 17:56:15 +00:00
Markus Lehtonen d8137be4de bitbake: cache: Make BB_DONT_CACHE variable external
This makes it possible to prevent a recipe to be cached, and thus,
parsed every time.

Use with care.

[YOCTO #8853]

(Bitbake rev: 78335c1fbe5266116700c2413aac28b00423a75b)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-23 17:54:41 +00:00
Michael Wood 3025092142 bitbake: toaster: orm Add a constant for the CustomImageRecipe's layer name
Use a constant to define the name for the toaster custom images layer;
this constant is then used to identify this layer in various places.

(Bitbake rev: 2540969ec71612af7f9041cadcc401513e9b357b)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-23 17:53:55 +00:00
Richard Purdie 8037ba4d86 bitbake: bb/tests/fetch: Update cups url
Update the upstream url used for testing cups versions after upstream website
changes.

(Bitbake rev: 5f06041d4936fc22297945bbbad7020bfa9083c6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-21 12:43:12 +00:00
Brendan Le Foll 3c66619d50 bitbake: fetch2/npm: fix ud.registry so that alternative registries can be handled
Fixes [YOCTO #9231]

npm when given an invalid registry URL with --registry actually goes and
fetches from the default registry, but this commit makes sure it goes to the
specified one.

(Bitbake rev: 7c849be7c70a5db4f66fe3041486abb923b5e4ee)

Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-12 09:35:52 +00:00
Lucas Dutra Nunes d53413d3a8 bitbake: server/process: Try connecting 4 times before giving up
Instead of trying one time with a timeout of 20 seconds try 4 times with
a timeout of 5 seconds, to account for a slow server start.

(Bitbake rev: 4a7fe63126dd8177baa5ad21e59e0bebeea8c596)

Signed-off-by: Lucas Dutra Nunes <ldnunes@ossystems.com.br>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 22:47:29 +00:00
Elliot Smith 0dcab0258e bitbake: toaster: rework task buildstats storage and display
The data available from buildstats is now more fine grained than
previously, so take advantage of that to enrich the data we save
against tasks:

* Store the CPU usage for user and system separately, and display
them separately.
* Disk IO is now measured in bytes, not ms. Also store the
read/write bytes separately.
* Store started and ended times, as well as elapsed_time. This
will enable future features such as showing which tasks were
running at a particular point in the build.

There was also a problem with how we were looking up the Task
object, which meant that the buildstats were being added to
new tasks which weren't correctly associated with the build. Fix
how we look up the Task (only looking for tasks which match the
build, and the task and recipe names in the build stats data) so
the build stats are associated with the correct task.

[YOCTO #8842]

(Bitbake rev: efa6f915566b979bdbad233ae195b413cef1b8da)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 22:45:16 +00:00
Peter Kjellerstedt aebc22dbfa bitbake: fetch2: Make SRC_URI[md5sum] and SRC_URI[sha256sum] expand their values
For some reason, the values for SRC_URI[md5sum] and SRC_URI[sha256sum]
were not being expanded. That lead to the following code not working
as expected:

SRC_URI = "http://.../${PN}-${PV}.tar.gz"

MD5SUM = "123abc..."
SHA256SUM = "abcd1234..."

SRC_URI[md5sum] = "${MD5SUM}"
SRC_URI[sha256sum] = "${SHA256SUM}"

(Bitbake rev: ba011470df0ea8bd89f01c0b02ec4b3969e60ce7)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 22:45:16 +00:00
Ed Bartosh d405f97af3 bitbake: xmlrpc: fix bug in setting XMLRPCServer.single_use
XMLRPCServer.single_use attribute was always set to False.
This caused xmlrpc server to keep running after build is done as
BitBakeServerCommands.removeClient only shuts down server if its
single_use attribute is set to True.

(Bitbake rev: 0a60b0928a0a746a60d2c2f294ff1903963c7086)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 22:45:16 +00:00
Paul Eggleton c50bdb300b bitbake: fetch2/npm: add missing URL argument to ParameterError
Without this you get a rather odd traceback instead of the proper
exception message.

(Bitbake rev: 2fe1826d3077eeda6cde433d3a1e6620f74e08dd)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 22:45:16 +00:00
Paul Eggleton fbf27c4c5d bitbake: fetch2/npm: properly handle npm dependencies
The output of "npm view dependencies" isn't entirely JSON if there are
multiple results, but the code here was just discarding the output if
the entire thing didn't parse as JSON. Split the output into lines and
iterate over it, parsing JSON fragments as we find them; this way we end
up with the last package's dependencies since it'll be last in the
output.

Digging further, it seems that the dependencies field reported by "npm
view" also includes optional dependencies. That wouldn't be a problem
except some of these optional dependencies may be OS-specific; for
example the "chokidar" module has "fsevents" in its optional
dependencies, but fsevents only works on MacOS X (and is only needed
there). If we erroneously pull in fsevents, not only is it unnecessary
but it causes "npm shrinkwrap" to throw a tantrum. In the absence of a
better approach, look at the os field and discard the module (along with
any of its dependencies) if it isn't for Linux.

As part of this, we can reduce the calls to npm view to one per package
since we get the entire json output rather than querying twice for two
separate fields. Overall the time taken has probably increased since we
are being more thorough about dependencies, but it's not quite as bad as
it could have been.

(Bitbake rev: 436d67fe7af89ecfbd11749a6ae1bc20e81f2cc8)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 22:45:16 +00:00
Paul Eggleton ef6a4518e4 bitbake: fetch2/npm: fix errors with some version specifications
"2 || 3" is a valid version specification for a dependency in an npm
package.json file, but of course that looks like something else when
sent to a shell. Quote the version value to avoid this.

(Bitbake rev: bea0246831a46d943d2e27d6b38f6e498bd3413c)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 22:45:16 +00:00
Ed Bartosh 3322fa7670 bitbake: toasterui: fix warning 'Unknown event'
Continue after processing BuildStarted event to fix
WARNING: Unknown event: <bb.event.BuildStarted object at 0x2554150>

(Bitbake rev: 12f1fb8c9b70fea0c9145f881bcceb8af32df6af)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:04 +00:00
Ed Bartosh 621cbc86c9 bitbake: toasterui: exit on final events
Toasterui exits only if bitbake observer shuts down.
In build mode it should exit when build is done.

Made toasterui exit on bb.command.CommandCompleted,
bb.command.CommandFailed and bb.command.CommandExit events
when it's running in build mode.

(Bitbake rev: b11f9d6d3c2eb615335901e1dcea699daf3afb4c)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:04 +00:00
Ed Bartosh 8e138b7f2a bitbake: toasterui: make toasterui to work in build mode
Currently toasterui works only in observer mode. This is
artificial limitation which was made to support current toaster
design. As we decided to stop using bitbake server we'll
need to run toasterui also in build mode.

[YOCTO #7880]

(Bitbake rev: d4b5796899c3ca5c7becd7322291afd8afb35a31)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:04 +00:00
Ed Bartosh 0a6130697e bitbake: toasterui: check if setEventMask succeeded
Currently toasterui ignores return value of setEventMask
command, which created confusing difference between set of
events set by this command and the real set used in the code.

Checked if setEventMask succeeded. Print error message and
exit if it's not.

(Bitbake rev: 6e3f13ffb47102b5df2da91fbc3f5da3179245b2)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:04 +00:00
Ed Bartosh ac941ac0a2 bitbake: command: make setEventMask readonly
Executing setEventMask command when bitbake server is in readonly
mode causes runCommand to fail with the following error:
 'Not able to execute not readonly commands in readonly mode'

Set readonly attribute for setEventMask command to make it working
for Toaster UI. This should not do any harm as this command doesn't
influence cooker state.

(Bitbake rev: 8a47d30b2555255fbf6049c5ed69b29664c32b17)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:04 +00:00
Ed Bartosh dd3da9aa47 bitbake: toasterui: update list of events
Removed events not used in the code from the list.
Added events that are used in the code.

(Bitbake rev: 16b14ec16049cc2040a60ad5fc95f6e19dda91a6)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:04 +00:00
Ed Bartosh f56fa5dfd4 bitbake: toasterui: reformat list of events
Reformatted and reordered list of events to make changes
easily and see them clearly in the diffs.

(Bitbake rev: 42a2d1115f2b23dc063a3172285ca3be73cf70bb)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:04 +00:00
Paul Eggleton a54cebe929 bitbake: fetch2/npm: ignore unknown headers in tarballs
Tarballs that are fetched down via npm repositories seem to often have
unknown headers. This doesn't affect our ability to extract the contents
though so we don't really care to see those warnings.

(Bitbake rev: b38975103e52a0c25e9ad9032c8cca1c47cbdcc2)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:03 +00:00
Paul Eggleton 0cd1be1f09 bitbake: fetch2/npm: handle alternative dependency syntax
npm allows you to specify a dependency as a Github username+path, or
omit the version entirely. You can hit these if you don't use a
shrinkwrap file, with the result that the code later fails due to the
output of "npm view" being empty; so handle this lazily by just ignoring
this part of the dependency if it's not really a version.

(Bitbake rev: 7b7a65c44dbdd5ba9366d4e2093f76df8758d546)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:03 +00:00
Paul Eggleton d9999279d9 bitbake: fetch2/npm: fix indentation
No code changes, just fix to use four spaces.

(Bitbake rev: 66a9ee7d54ca9c25209f72da079f260ccdcc872a)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-07 17:23:02 +00:00
brian avery a7d1b95c96 bitbake: ui: remove the puccho ui
This ui does not work in master, nor has it been updated for several years.

[YOCTO #9178]

(Bitbake rev: 9fad1d13eed1f725971e6d12d3977cd31e07019a)

Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-02 22:41:23 +00:00
bavery a9dc72f08b bitbake: hob: removal of hob ui and associated ui files
We've been gearing up the Toaster web UI to replace the Hob (GTK+ based) UI
for some time now; Hob has basically been on life support for the past few
releases. As of late last month in master, Toaster has the capability to
select the packages in an image, removing the last thing that Hob could do
that Toaster couldn't.

To recap, the reasons why Hob is being removed include:

- The code is tightly woven into BitBake, making it fragile. This means it
needs significant QA and maintenance on an ongoing basis.

- Some of the implementation is not ideal; we'll be able to remove some cruft
from BitBake and OE-Core at the same time.

- It's GTK+ 2 based, not the current GTK+ 3.

- Toaster is now a much more capable UI and is being actively maintained

The discussion about removing hob can be found at:
http://lists.openembedded.org/pipermail/openembedded-architecture/2016-February/000082.html

(Bitbake rev: be2cceea159c6ca9111eff3df87b98513eab6d72)

Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-02 22:41:23 +00:00
Brendan Le Foll 27468dbc3a bitbake: fetch2/npm: Add missing ParameterError import
(Bitbake rev: b583a40c9086b3587065995d8cee45b7ea36c1b3)

Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-02 22:41:22 +00:00
Brendan Le Foll 44e3461969 bitbake: npm: in cases where shrinkwrap resolved a git URL, ignore it and grab dist.tarball
npm-shrinkwrap will sometimes resolve a git URL which instead of a http url, in
this case go and grab the dist.tarball via npm instead of using the resolved
URL.

(Bitbake rev: eb53b927ff59aa19cf28bc46beb9f9a185a59990)

Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-02 22:41:22 +00:00
Richard Purdie 2a7318133b bitbake: fetch2: Fix unpack for absolute file urls
The previous commit breaks absolute pathnames in file:// urls, this
fixes it.

(Bitbake rev: b8113a1800687a37a26ac28deafdbafd74cc138e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-02 22:41:22 +00:00
Alexander Shashkevich 865d2feff6 bitbake: fetch2: fixes copying of file://dir; subdir=foo, bug 6128 and bug 6129
When in SRC_URI appears file://dir;subdir=foo unpacker copies 'dir' to ${WORKDIR}, not
${WORKDIR}/foo as it should be.

These changes are fixing following bugs as well:
Bug 6128 - Incorrect wildcard unpack behaviour in fetcher
Bug 6129 - Local directories unpack to a different location than local files

(Bitbake rev: e659a3b0c2771679057ee3e13cd42e6c62383ff2)

Signed-off-by: Alexander Shashkevich <alex@stunpix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-02 22:41:22 +00:00
Richard Purdie 2e27c4bf23 bitbake: fetch2/npm: Enable fetcher
For some reason the enablement piece of the patch went missing, add it.

(Bitbake rev: 0270b5a3873ed0aeca3a66198c87a6164fb644b8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 12:01:07 +00:00
Richard Purdie 030d920e2d bitbake: providers: Fix PREFERRED_VERSION lookup for '_' in PN
PN can contain '_', e.g. gcc-cross-x86_64 and an override cannot
hence we do this manually rather than use OVERRIDES.

(Bitbake rev: 7a6baf02617d1edced4eaff235e73d746e2a3b68)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:34:38 +00:00
Richard Purdie c679a3dd41 bitbake: fetch2: Skip lockfiles and donestamps for local files
For local files, there are no races with downloads, we don't need ".done"
stamps and we don't need lockfiles.

This considerably cleans up DL_DIR and all the pointless ".done" files
as well as removes stalls over local files with the same name.

(Bitbake rev: 48e903745db578d9b9b425a8d411c1369df0eb94)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:34:38 +00:00
Richard Purdie d01042eda9 bitbake: fetch2/__init__.py: Error if lockfile path invalid
Rather than create ".lock" and ".done" files with no name, error,
forcing us to fix the cases where this is a problem.

(Bitbake rev: 81158071508cc68c39db7d501370872f44d335cc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:34:38 +00:00
Richard Purdie ab7b7bf94d bitbake: fetch2/__init__: Fix decodeurl to better handle urls without paths
If we specify urls such as npm://somehost;someparams the fetcher currently
does a poor job of handling mirrors of these urls due to deficiencies in the
way decodeurl works. This is because "somehost" is returned as a path, not
a host.

This tweaks the code so that unless its a file url, the host is returned
correctly.

This patch also adds test cases for these urls to the exist set of test
urls.

We need to tweak the URI() class since this thinks this is a relative url
which is clearly isn't. We also need to handle the fact that encodeurl will
error if passed a url of this form (it would want the path to be '/'.

(Bitbake rev: 83203cd2e677706e0111892a7843b83263cb8bd9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:34:38 +00:00
Richard Purdie 06b4d8ffe4 bitbake: fetch2/wget: Set localfile for directories
If we wget something which looks directory like we end up with lock files
and done stamps without names, they also all use the same lockfile.

This change ensures that we use separate lock files based on the url
and avoid creating the mysterious ${DL_DIR}/.done files.

(Bitbake rev: 20bc82086018832e047345a672d74b6c1c113650)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:34:38 +00:00
Richard Purdie 12ade9b093 bitbake: fetch2/npm: Add mirroring support for npm fetcher
(Bitbake rev: bff46c614d3f9cc18a1c8908c47842712e0e3a8c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:09 +00:00
Brendan Le Foll ca5b6d66e0 bitbake: fetch2/npm: Add npm fetcher
npm fetcher with support for shrinkwrap files and lockdown files to easily
download and install an npm package with strict dependency resolution.

The SRC_URI should be in the format of:
SRC_URI = "npm://registry.npmjs.org/;name=${PN};version=${PV}"

To add a shrinkwrap and lockdown file use:
NPM_SHRINKWRAP := "${THISDIR}/${PN}/npm-shrinkwrap.json"
NPM_LOCKDOWN := "${THISDIR}/${PN}/lockdown.json"

(Bitbake rev: dec75bbc5d075acb322dad8b1c40d6bd518dc9fd)

Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:07 +00:00
Brendan Le Foll 813bd1f806 bitbake: utils.py: Add sha1_file call
This is useful as npm-lockdown uses sha1 because npm releases the sha1 of
packages and whilst this is undocumented it seems no other algorithm is
supported

(Bitbake rev: fd5d9011f6dd7029895b64d8a02d33185b9aa8ae)

Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:05 +00:00
Richard Purdie 4e5e501155 bitbake: data_smart: Drop default expand=False to getVarFlag [API change]
At some point in the future, getVarFlag should expand by default. To
get there from the current position, we need a period of time where the
expand parameter is mandatory.

This patch starts that process. Clear errors will result from any code
which doesn't provide this. Layers can be fixed with an expression
like:

sed -e 's:\(\.getVarFlag([^,()]*, [^,()]*\)):\1, False):g' -i `grep -ril getVar *`

(Bitbake rev: aa3faebdf6af66ab34f74d328b2113de0b08c7ee)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:48:00 +00:00
Richard Purdie c7610aa353 bitbake: data_smart: Drop default expand=False to getVar [API change]
At some point in the future, getVar should expand by default. To get
there from the current position, we need a period of time where the
expand parameter is mandatory.

This patch starts that process. Clear errors will result from any code
which doesn't provide this. Layers can be fixed with an expression
like:

sed -e 's:\(\.getVar([^,()]*\)):\1, False):g' -i `grep -ril getVar *`

(Bitbake rev: fab717d303df0bcef737661f6917f275f35215a4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:48:00 +00:00
Markus Lehtonen 4f0ab27c6a bitbake: SignatureGeneratorBasic: make checksum cache file configurable
Define a new bitbake configuration variable BB_HASH_CHECKSUM_CACHE_FILE
that can be used to define the cache file to use for file checksum
cache.

(Bitbake rev: a965b390d6240e279c190b92b17c0573e9bd604c)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:41:16 +00:00
Markus Lehtonen 0cdf1934fd bitbake: MultiProcessCache: make cache filename configurable
If no cache file name is given a default from class variable is used,
like before.

(Bitbake rev: 2602a312818f564961de7dfa63c429d45ff9e5ac)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:41:16 +00:00
Markus Lehtonen ca552bb4b1 bitbake: FileChecksumCache: add get_checksums() method
Move the local file checksum functionality from bb.fetch2 into
bb.checksum module.

(Bitbake rev: 4f60933283f377d68f191db849dac6c1dc7a0aed)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:41:16 +00:00
Markus Lehtonen 8f61f2d881 bitbake: bb/runqueue: save task file dependency cache onto disk
Before this patch the usage of cache was quite useless as the file
checksums were not actually cached on disk but re-calculated every time.
This patch utilises the new writeout_file_checksum_cache() method of the
SignatureGenerator class to do the job.

(Bitbake rev: 5ac9cbf405841ed3f65e6f99a3cee032567fb182)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:41:16 +00:00
Markus Lehtonen 5177b1e4ac bitbake: SignatureGenerator: add method for saving the file checksum cache
Extend the API in order to be able to write out the file checksum cache
onto disk. SignatureGeneratorBasic class now implements a method that
update the fetcher local files checksum cache with the task file
dependency checksums.

(Bitbake rev: ecdabd321d48fa367b89ebffc00aa525b6eaa95c)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:41:15 +00:00
Markus Lehtonen 97617fd675 bitbake: bb/cache: drop some unused arguments
Drop unused 'd' argument from the cache save methods, simplifying the
API.

(Bitbake rev: 81bc1f20662c39ee8db1da45b1e8c7eb64abacf3)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:41:15 +00:00
Mirela Rabulea 5a87d8c477 bitbake: Allow Hob to run images on a custom simulator, other than qemu
The current behavior of Hob is that there is a "Run Image" button which becomes visible only for qemu images.

My suggested change is:
- if an image is selected and it is qemu-compatible, let the "Run image" button be named "Run qemu image"
- if an image is selected and it is not qemu-compatible, let the same button show up with the name "Run custom image", and besides that, an option shows-up to allow the selection of the custom script (by default it points out to runqemu script) to be used for launching this custom image

Note: in case there is more than one toggled image (qemu runnable or deployable), when the user clicks the "Run custom image" button, a dialog will be presented, allowing to choose between any of the existing images.

[YOCTO #8940]

(Bitbake rev: cc4cfc2370297b8feb2dc39d4262e73adf06c09a)

Signed-off-by: Mirela Rabulea <mirela.rabulea@nxp.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18 07:41:15 +00:00
Christopher Larson a25858982a bitbake: bb.ui.knotty: prefix task messages with recipe/task
An example prefix: `perl-5.22.1-r0 do_compile:`

(Bitbake rev: 792b759e59e31d2e43d525a6e50d866b4f51f072)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-16 09:04:23 +00:00
Christopher Larson 4bf8b21ab8 bitbake: Move bb.{debug,note,..} into their own logging domain
This lets us filter and use -l to show messages from that source specifically.

(Bitbake rev: 7946927156dec33364418988eb921ddb273660eb)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-16 09:04:23 +00:00
Richard Purdie 8b217208cf bitbake: knotty: Set exit failure code on runQueueTaskFailed events
If the worker segfaults, we may never see a TaskFailed event from it, only
a runQueueTaskFailed event. In this case, return_value isn't getting set
leading to an incorrect exit code from bitbake. Fix by setting return_value
in both places.

(Bitbake rev: e5dd50e0d95d532fe31dde61f8c6b1a7a72321e9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-15 16:48:12 +00:00
Richard Purdie a9223e20d1 bitbake: taskdata: Fix traceback issue with missing provider
If there is a missing provider and we're using "-k" mode alongside "-w",
we could get a traceback since there was no provider. Add tests to avoid this.

(Bitbake rev: 90a4805e4e770a433b4394ea99792731e9a4b546)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-15 16:48:12 +00:00
Richard Purdie 7593756cbb bitbake: cooker: Improve cache handling
We're seeing issues where the self test, which uses tinfoil doesn't
notice the changed contents of include files. The issue is
cached_statements in the parser being reused when the files have changed.

Whilst looking at this, I realised there were some other issues:

* We need to also invalidate the mtime cache when cooker restarts
* We should pass full filenames to the file invalidation code
* We should process cached_statements as part of inotify invalidation

With these fixes, the caching is more reliable for memory resident
bitbake too. It does raise some questions about cache validation and
lifecycles and indicates bitbake does need more work in the area,
preferably with the removal of the globals. This at least highlights
and works around some of the current issues.

(Bitbake rev: 3f507ff8bc467fba936cf3f31bb8ea8e02f168e8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-15 16:48:12 +00:00
Richard Purdie 1c2d632a02 bitbake: Bump version post release to 1.29.0
(Bitbake rev: da1107a2c2331b3335c7bd46821edcb1933a0ff8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-12 13:44:22 +00:00
Ed Bartosh f9f93ae856 bitbake: cooker: gracefully shutdown parsers
CookerParser.shutdown code doesn't do all required work to shutdown
parser processes. As a result bitbake hangs if interrupted during
parsing. Putting None into the parser_quit queue should fix this issue
as it makes parsers to quit main loop.

(Bitbake rev: f67307977e8f089ce6d208d3e9de2a6a1768757e)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11 22:34:19 +00:00
Elliot Smith 1f7f077466 bitbake: buildinfohelper: unset brbe variable when build finishes
The brbe variable is set on the bitbake server when the Toaster
UI starts. This enables Toaster to associate events with the
correct build and build environment.

However, the brbe variable is also used when a build starts to
identify whether a new build needs to be created, or an existing
one looked up. This causes a bug for command-line builds which
happen after a Toaster-triggered build: because the brbe variable
is never unset on the server or the buildinfohelper, the new
command-line build events are treated as originating from the
previous build.

Ensure the brbe variable is reset when the buildinfohelper "closes"
a build, so that each build then either sets the brbe variable
(Toaster-triggered builds) or leaves it blank (command-line builds).

Also modify the localhostbecontroller so that the brbe variable
is not set on the server and not looked up from the server. This
ensures that it is only set when the triggerBuild() method is
called, and that it remains as None for command-line builds.

[YOCTO #9021]

(Bitbake rev: 4a6a8d0074f62208d843b06344be31ae73d9b745)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11 22:34:19 +00:00
Richard Purdie 400f53003e bitbake: build: Improve python execution tracebacks
If something fails in a exec_func_python() the current stack trace shows
incorrect filenames and linenumbers. For example:

The stack trace of python calls that resulted in this exception/failure was:
File: '/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb', lineno: 200, function: <module>
     0196:                chksum = bb.utils.sha256_file(fn)
     0197:                f.write('%s\t%s\n' % (chksum, os.path.relpath(fn, baseoutpath)))
     0198:
     0199:
 *** 0200:copy_buildsystem(d)
     0201:
File: '/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb', lineno: 9, function: copy_buildsystem
     0005:IMAGE_FEATURES += "splash package-management x11-base x11-sato ssh-server-dropbear hwcodecs"
     0006:
     0007:LICENSE = "MIT"
     0008:
 *** 0009:inherit core-image
     0010:
     0011:IMAGE_INSTALL += "packagegroup-core-x11-sato-games"
File: '/usr/lib/python2.7/subprocess.py', lineno: 535, function: check_call
     0531:    The arguments are the same as for the Popen constructor.  Example:
     0532:
     0533:    check_call(["ls", "-l"])
     0534:    """
 *** 0535:    retcode = call(*popenargs, **kwargs)
     0536:    if retcode:
     0537:        cmd = kwargs.get("args")
     0538:        if cmd is None:
     0539:            cmd = popenargs[0]

The problem is the use of "FILE" to obtain the current filename. Instead,
we therefore inject the function being executed into the methodpool which
allows us to correct its linenumber and filename information. We can then
clearly mark the initial piece as autogenerated and the rest of the linenumber
and filename information should be correct. Afterwards the trace starts:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:copy_buildsystem(d)
     0003:
File: '/media/build1/poky/meta/classes/populate_sdk_ext.bbclass', lineno: 66, function: copy_buildsystem
     0062:    import glob
     0063:    import oe.copy_buildsystem
     0064:    import subprocess
     0065:
 *** 0066:    subprocess.check_call("foo")
     0067:
     0068:    oe_init_env_script = d.getVar('OE_INIT_ENV_SCRIPT', True)
     0069:
     0070:    conf_bbpath = ''
File: '/usr/lib/python2.7/subprocess.py', lineno: 535, function: check_call
     0531:    The arguments are the same as for the Popen constructor.  Example:
     0532:
     0533:    check_call(["ls", "-l"])
     0534:    """
 *** 0535:    retcode = call(*popenargs, **kwargs)
     0536:    if retcode:
     0537:        cmd = kwargs.get("args")
     0538:        if cmd is None:
     0539:            cmd = popenargs[0]

We can't inject into methodpool at parsing time, since there may be
_append or other override operations against the function before its
execution.

(Bitbake rev: fae153095d23157dd7e72c29f683f86149ee33a8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 15:47:11 +00:00
Richard Purdie aece748762 bitbake: build/data: Don't expand python functions before execution [API change]
Right now, if you have some python code like:

X = "a"

def somefunction(d):
    d.setVar("X", "b")
    d.setVar("Y", "${X}")

then any sane person would expect that Y = "b" at the end of the
function. This is not the case, Y = "a".

This is due to the python function being expanded before execution, the
executed code would read d.setVar("Y", "a"). This understandably
confuses people, it also makes it near impossible to write ${} in a
python function without unintended things happening.

I think there is general agreement we should fix this and standardise
on non-expansion of python functions. We already don't expand anonymous
python (mostly).

I've checked OE-Core with buildhistory before and after this change and
there were a small number of issues this exposed which I've sent
patches for.

I propose we default to not expanding python code and then deal with
any consequences from that if/as/where identified. This will improve
new user understanding and usability of the system, it also allows
several long standing weird expansion issues to be fixed.

(Bitbake rev: 8bf33a8e92c0e188fa392030025756196c96fcbb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 15:47:11 +00:00
Richard Purdie e39cfb1f9c bitbake: cooker: Don't expand python functions in variable dumps
We don't want to expand python functions since they aren't expanded
at execution time (e.g. anonymous python). They can also have side
effects.

This function is primarily used by toaster for variable dumps for later
display. The lack of expansion of python functions won't matter in this case
and actively helps some variable handling (e.g. SRCPV).

(Bitbake rev: 3f5520b4844a4bdd615046479ba08ed192bdc8cd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 15:47:11 +00:00
Richard Purdie f652b6b9d0 bitbake: data: Don't expand python functions for variable dependencies
Expanding python functions for variable dependencies doesn't really make sense,
not least since this causes execution of any inline python, it also makes it
impossible to write expressions like d.expand("${X}") of d.setVar("X", "${Y}")
which may have the wrong values if expanded now.

This starts to standardise the approach across bitbake for handling python code.

(Bitbake rev: 765a2480dbe288f64562a9611dd93b6b6dd0a64e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 15:47:11 +00:00
Richard Purdie d3e0c448ea bitbake: data_smart: Avoid expanding anonymous python functions
We don't expand anonymous python before execution, so nor should
we do this when calculating checksums for them.

(Bitbake rev: 5f10987edda35b08970a6dd6ccf9febad271ce3e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 15:47:11 +00:00
Richard Purdie 99184d7879 bitbake: BBHandler/ast: Merge handMethod and handleMethodFlags
The functionality overlap between these two functions is significant and
its clearer to handle both things together since they are intimately
linked. There should be no behaviour change, just clearer code.

(Bitbake rev: 391aa4afc91be90d8d3ee47e1bf797d6ebe61a71)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:32:46 +00:00
Richard Purdie 6ba69b4383 bitbake: utils: Drop datastore function inspection during exception
When we use functions from the data store, they now have correct line number
and filename information. This function would attempt to correct line numbers
which doesn't need correcting, leading to misleading messages to the user.
Therefore remove this code as being obsoleted.

(Bitbake rev: 918bec86bc8ee94feb82380ff410d9fdcbe9e720)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:32:46 +00:00
Schroeder, Henning f8a44b160d bitbake: cooker: extended dot styling
Extended the dot styling of dependencies created by bitbake -g in dot syntax to differentiate between the various kinds.
  depends: solid
  rdepends: dashed
  rrecommends: dotted

The change observed is that depends get an explicit style which is the same as dot default behavior and the runtime recommends get
dotted while before they were dashed. This helps to distinguish them graphically as well as eases post processing by script.

(Bitbake rev: 86e78e0ca7aa5452411f35239942ecee3d8824ec)

Signed-off-by: Henning Schroeder <henning.schroeder@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:32:46 +00:00
Michael Wood a8ab1c6b25 bitbake: toaster: buildinfohelper CustomImagePackage update dependency info
Instead of keeping the original dependency information for the pool of
CustomImagePackage reset it with each new build.

(Bitbake rev: a0b97ffc7a468bad081ce3276c74728bf6830250)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:21 +00:00
Michael Wood a6e4f94b01 bitbake: toaster: buildinfohelper Add the concept of CustomImagePackage
This adds the concept of CustomImagePackage this is similar to the way
layers and recipes work in that we have a set of data which is part of
the build history and a set of data which is part of the configuration
data that toaster uses to guide people in configuring their project. We
create a set of  built_packages for every build but only create a package
for configuration purposes if we don't already have one, so that the
CustomImagePackage only ever contains a unique list of packages that are
available to be added and removed from a CustomImageRecipe.

(Bitbake rev: f81bb65883baa6c0f8a4d48a4de3291a10543992)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 13:29:17 +00:00
Aníbal Limón d0d85a4d6b bitbake: bb/fetch2: Move export_proxies function from wget to utils.
In order to use in other modules since is a common function
when needs to get proxies working.

(Bitbake rev: 85c529044381895556d603a3974de22392646a22)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-08 10:52:40 +00:00
Ed Bartosh 916c73dfef bitbake: cooker: shutdown cooker parser on shutdown
Currently any not catched exception in cooker causes bitbake
to hang because of not terminated children of CookerParser.

Long term solution would be to reimplement Cooker as a context
manager and terminate parser children in its __exit__ method.

Partial fix is to call CookerParser.shutdown in Cooker.shutdown in
hope that all Cooker exceptions are caught and shutdown method is
called.

[YOCTO #8900]

(Bitbake rev: 3f67600dc3292bc8208644ce89e8bf7ab95cf2e7)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-06 23:13:47 +00:00
Richard Purdie 8857498ba4 bitbake: fetch2/osc: Clean up old variable syntax
Since people do copy and paste these things, clean up old syntax styles.

(Bitbake rev: 4fb028b0bd14d3e4b3fd7a89c643528728566476)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-06 23:13:47 +00:00
Richard Purdie 54da82972c bitbake: fetch2/osc: Remove hardcoded url
This shouldn't be in here, use a variable instead.

(Bitbake rev: 2e25d09a1ab62ccc3573d13114d59838cf4b07f2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-06 23:13:47 +00:00
Richard Purdie 6ae662acdc bitbake: parse/ast: Mark anonymous functions as python functions
Anonymous functions are python functions, set the variable
flags as such so we can detect them and avoid expansion where
needed.

(Bitbake rev: 1b303785c578bbae3a89be8d751d80fba860f62e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-04 23:47:49 +00:00
Richard Purdie 9913fd88d9 bitbake: codeparser: Improve handling of data.expand() dependencies
Currently bitbake doesn't parse into data.expand() expressions,
relying on high level expansion of python code to handle this.

One of the tests does however test this works.

We don't really want to be doing string expansion on python code,
so specifically parse into expand() function calls so that when
the high level behaviour is tweaked, the self tests continue to
pass and that we do continue to handle expand() function calls as
best we can.

(Bitbake rev: b12c17be5e4a74c9680876605c87f46501f78d28)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-04 23:47:49 +00:00
Richard Purdie 4628fe12e7 bitbake: lib/bb: Add expansion parameter to getVarFlag
This sets the scene for removing the default False for expansion from
getVarFlag. This would later allow True to become the expand default.

On the most part this is an automatic translation with:

sed -e 's:\(\.getVarFlag([^,()]*, [^,()]*\)):\1, False):g'  -i `grep -ril getVar *`

There should be no functional change from this patch.

(Bitbake rev: 7c3b99c6a716095af3ffce0b15110e91fb49c913)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-04 23:47:49 +00:00
Felipe F. Tonello b98866d003 bitbake: fetch2/gitsm: Fix when repository change submodules
This fix a problem when checking out a commit that changes the submodules
previously checkout.

Example:
Recipe uses branch A and then it updates to use branch B, but branch B has
different submodules dependencies then what branch A previously had.

(Bitbake rev: 54a3864246f2be0b62761f639a1d5c9407aded4f)

Signed-off-by: Felipe F. Tonello <eu@felipetonello.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-04 23:47:49 +00:00
Richard Purdie 390c2c1268 bitbake: data_smart: Add missing expand parameter to getVar call
(Bitbake rev: 2876019e696ff2af164961d5d4c8e3ea9dfefc23)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-04 23:47:49 +00:00
Phil Blundell a4a5d1f323 bitbake: cooker, bitbake-worker: Fix spelling of "received"
I before E, except after C...

(Bitbake rev: 14c9593265f7469cb8a205a46f845ac7491246df)

Signed-off-by: Phil Blundell <pb@pbcl.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-01 11:20:42 +00:00
Richard Purdie 8f6b9c75a7 bitbake: cooker: Only start as many parse threads as we need
If we're only going to parse one recipe, no point in starting
a large number of threads.

(Bitbake rev: b977faf59dc08050a44a16032fe52d1bbb80f2a1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-01 11:20:42 +00:00
Richard Purdie 602da7c921 bitbake: knotty: Don't show errors for universe provider issues
When running universe builds, we don't expect an error exit code for
provider warnings. Change the error messages to warnings in this case.

This deals with errors causing problems on our autobuilders amongst
other issues.

(Bitbake rev: d4989fb0355476de172169f0698757f7360e9a1f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-01 11:20:42 +00:00
Sujith H 3072361f86 bitbake: bitbake: BBUIHelper: Remove function findServerDetails
findServerDetails function can be removed safely
from the source tree. Couldn't find any files
calling this function.

(Bitbake rev: 46871f769db13ccd36deedc5b6f3dbc0a3d31c4b)

Signed-off-by: Sujith H <sujith.h@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-30 11:43:56 +00:00
Peter Kjellerstedt 28c041c1e7 bitbake: fetch2: Simplify logic in verify_checksum()
The recent change to verify_checksum() to only show checksum warnings
if no checksums are supplied made it possible to simplify the logic a
bit more.

(Bitbake rev: 1dc00b874acae44bbba9d8028d94f7bc97ddcd76)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-30 11:43:56 +00:00
Richard Purdie 5375e6431c bitbake: bitbake: Set process names to be meaninful
This means that when you view the process tree, the processes
have meaningful names, aiding debugging:

$ pstree -p 30021
bash(30021)───KnottyUI(115579)───Cooker(115590)─┬─PRServ(115592)───{PRServ Handler}(115593)
                                                ├─Worker(115630)───bash:sleep(115631)───run.do_sleep.11(115633)───sleep(115634)
                                                └─{ProcessEQueue}(115591)

$ pstree -p 30021
bash(30021)───KnottyUI(117319)───Cooker(117330)─┬─Cooker(117335)
                                                ├─PRServ(117332)───{PRServ Handler}(117333)
                                                ├─Parser-1:2(117336)
                                                └─{ProcessEQueue}(117331)

Applies to parse threads, PR Server, cooker, the workers and execution
threads, working within the 16 character limit as best we can.

Needed to tweak the bitbake-worker magic values to tell the
workers apart.

(Bitbake rev: 539726a3b2202249a3f148d99e08909cb61902a5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-30 11:43:56 +00:00
Richard Purdie 5b234d1539 bitbake: utils: Add ability to change the process name
Being able to tell the bitbake processes apart is useful for debugging.
Add a helper function which allows this without making it a hard
dependency. Errors are ignored, this is just nice to have.

(Bitbake rev: fd7f1a94d196b8a3c445e313d9e699b352b1da97)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-30 11:43:55 +00:00
Andre McCurdy 0b06924759 bitbake: data.py: avoid double newlines at the end of functions in emit_var()
A newline is always appended to the function body when it's written
out, so strip any trailing newlines which may be there already.

(Bitbake rev: 8a3f50936113e15d2f2822f6aee494204fa1c24f)

Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-30 11:43:55 +00:00
Andre McCurdy 68600aee2d bitbake: build.py: minor shell_trap_code() formatting tweaks
Fix quoting of $BASH_COMMAND and avoid wrapping at 80 columns (the
script which follows is likely to contain some very long lines, so
line wrapping in bb_exit_handler() looks somewhat out of place).

(Bitbake rev: 8e12c8f8441a7c6a03e603c5789d6037945704c1)

Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-30 11:43:55 +00:00
Peter Kjellerstedt a948f5252a bitbake: cooker: Allow BBMASK to contain multiple regular expressions
Before, BBMASK was only permitted to contain one regular expression.
This made it hard to add to the BBMASK in multiple places as one was
supposed to separate the different regular expressions with a "|"
rather than with whitespace as is customary in BitBake variables.

Now one can specify any number of regular expressions in BBMASK. This
makes it possible to, e.g., mask out recipes in another layer from the
layer.conf file.

This also properly ignores any regular expressions that do not compile
(before an invalid regular expression would cause a ParseError in the
first bbappend file found stating that it was not a BitBake file...)

(Bitbake rev: 2c778ad50aceaffb855baf5f4aa0fed98c880870)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-29 17:03:53 +00:00
Ross Burton 9f22898e0f bitbake: fetch2/wget: fallback to GET if HEAD is rejected in checkstatus()
The core change here is to fall back to GET requests if HEAD is rejected in the
checkstatus() method, as you can't do a HEAD on Amazon S3 (used by Github
archives).  This meant removing the monkey patch that the default method was GET
and adding a fixed redirect handler that doesn't reset to GET.

Also, change the way the opener is constructed from an if/elif cluster to a
conditionally constructed list.

(Bitbake rev: 6ec70d5d2e330b41b932b0a655b838a5f37df01e)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-26 22:34:45 +00:00
Richard Purdie 4a5458f38f bitbake: fetch2: Don't show checksum warnings if a single checksum was supplied
If one checksum is supplied to a SRC_URI, we really don't want to show
warnings about the other type which isn't present as one checksum
is really good enough for most cases.

(Bitbake rev: 43358a9b595b2928458a5f463cf1949394160c3a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-24 09:40:42 +00:00
Paul Eggleton 76a281c870 bitbake: taskdata: add the ability to access world targets list
In certain circumstances it can be useful to get access to the world
targets list from a recipe in order to add dependencies on some or all
of the items in it. If a special function, 'calculate_extra_depends' is
defined in the recipe, and the recipe is to be built, then call it at
the right point before we calculate which tasks should be run. The
function can append items to the "deps" list in order to add
dependencies. This is not as tidy a solution as I would have liked, but
it does at least do the job.

As part of this change, the buildWorldTargets function was moved to
bb.providers to make it possible to call from taskdata.

Part of the implementation of [YOCTO #8600].

(Bitbake rev: aba0dce57c889495ec5c13919991a060aeff65d2)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-22 12:45:44 +00:00
Robert Yang 11a1f49630 bitbake: cache.py: check existence before add to cachedata.rproviders
The rprovides maybe contain duplicated lines when parse again, we need
check it before add to cachedata.rproviders, similar to what we had done
to cachedata.providers.

(Bitbake rev: 6c488afb0fe30a9655ec62a1d22f9f388365f012)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-22 12:45:44 +00:00
Robert Yang 05c17750c3 bitbake: taskdata.py: add RuntimeProviders to close matches
This is useful for newbie, for example:
$ bitbake rpm-build
ERROR: Nothing PROVIDES 'rpm-build'. Close matches:
  pm-utils
  rpm RPROVIDES rpm-build

[YOCTO #8881]

(Bitbake rev: 4b59eb8cc2321fe72f2988b6c9c0fecd4883255b)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-22 12:45:44 +00:00
Richard Purdie cf9cb65eec bitbake: data_smart: Don't show exceptions for EOL literals
If variables are unset, the code simply doesn't expand them, there
aren't errors. If the code is a python expression, this can get a bit
messy, see the attached test case. The python expansion code sees the }
of the unexpanded value rather than the close of the python expression
and then raises a SyntaxError exception.

Ideally, we'd update the code to match pairs of brackets. I don't know
how to do that with the current regex and this is unfortunately a
performance sensitive piece of code. We also run the risk of breaking
existing code in OE-Core where there are "{" characters but not "}"
to close them (PKGE and PE).

Rather than raising the exception, matching the existing "just return
the expression" behaviour seems more consistent with the standard
variable behaviour.

This addresses an issue found in the recent image.bbclass code where
there are some variables we choose not to expand (TMPDIR/DATETIME).

This patch also adds a test case for this behaviour. It wouldn't preclude
improved bracket matching code in the future either.

(Bitbake rev: d80d39e73223a50fda0090784303d2c57167bb4c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-22 12:01:24 +00:00
Olof Johansson c886a786cf bitbake: tests/codeparser.py: Add filename/lineno flags to test variable
A recent change in bitbake added filename/lineno information to the
parameters of bb.data.build_dependencies(). The codeparser tests
required a little adaption to the changes, adding the flags to the FOO
variable used in the tests.

The error seen when running the tests is a TypeError exception raised
in bb.codeparser:

  TypeError: int() argument must be a string or a number, not 'NoneType'

(Bitbake rev: f1fe674397ac5cd355696d5b4cc90b7cfa6c867f)

Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-19 17:38:51 +00:00
Richard Purdie f0d47a6d9a bitbake: Revert "runqueue.py: Ensure one setscene function doesn't mask out another which needs to run"
This reverts commit b22592af81.

That commit isn't entirely clear about why this change is needed but
I do have a usecase where this breaks things. If for example you run
"bitbake X -c packagedata" and that packagedata is in sstate, you'd
expect this to work.

If sstate doesn't contain a do_populate_sysroot for a dependency, you
would still expect the command above to succeed and you would not
expect
it to rebuild that dependency. With the current code, this isn't what
happens. The code finds the sstate for do_populate_sysroot missing,
this makes the task "uncovered" and this in turn makes it unskippable.

The example I found with this was avahi-ui, where it would trigger
a build of libdaemon to obtain its populate_sysroot.

Since this behaviour seems completely incorrect, revert the older patch
and we'll address any issues that crop up as a result.

(Bitbake rev: 36a9840a5da17cc14561881fdd6a4f2cb0a75e49)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-19 17:26:15 +00:00
Elliot Smith 68f3e1e067 bitbake: toasterui: log OSErrorException metadata events
OSErrors occurring in toaster.bbclass are converted to
OSErrorException metadata events. They were then being swallowed
as unprocessed events by toasterui, which made them difficult
to spot.

Explicitly catch OSErrorException events and log them so they
are easier to spot and debug.

(Bitbake rev: 69f2b2bc373ce114609600b59a6b6ccef20771c9)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-18 22:06:46 +00:00
Elliot Smith fb94754e83 bitbake: toasterui: listen for bb.event.MetadataEvent
The event mask for toasterui doesn't include MetadataEvents.
This means that we're missing the ArtifactFileSize event
(among others), which is the one we use to populate the SDK
artifact table.

Add that event type to the toasterui event mask so we can
record SDK artifacts as they are created.

[YOCTO #7603]

(Bitbake rev: d0276a831bb8cffd42c8367895633eaa1fa1ed30)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-18 22:06:46 +00:00
Richard Purdie f8508deb01 bitbake: Revert "fetch/git: Change to use clearer ssh url syntax for broken servers"
This reverts commit 4193e99adce8e88f12ac88d7578ad39575f7e346.

It seems the underlying issue was caused by ":" in the url which isn't
supported. The patch was therefore incorrect.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-15 15:02:40 +00:00
Richard Purdie da43a56d35 bitbake: Revert "fetch2/local.py: avoid using PREMIRROR"
This reverts commit e130dca85bac82bd4d88f94a6bf9fe36e8ad4d7c.

This is in fact a valid use case, for example the sstate.bbclass code
sets up SSTATE_MIRRORS as PREMIRRORS. Its quite common to map those
file:// urls to remote http:// urls and with the above change, this
no longer works.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-13 09:47:28 +00:00
Kristian Amlie 1724ffd32d bitbake: fetch2/git.py: Add missing "errno" module import.
This goes undetected most of the time, but when updating a repository,
if the ud.fullmirror file is not present, you end up getting an
exception instead of carrying on because the errno module is not
loaded (specifically "if exc.errno != errno.ENOENT").

(Bitbake rev: e6fca8480731ce817df9bee61438347a5e3d3017)

Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:28:55 +00:00
Chen Qi 74fa824e6c bitbake: bitbake: clean up stamp-base related codes
The 'stamp-base' and 'stamp-base-clean' related codes are no longer useful,
clean them up.

[YOCTO #8468]

(Bitbake rev: 7b4c42b315d4a26dd8f2ceb874a94737bf9f183e)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:28:55 +00:00
Richard Purdie 43328feccc bitbake: runqueue: Fix setscene task dependencies
Debugging suggests that setscene tasks are being a little greedy about their
dependencies, for example, lsof is insisting that gcc-runtime's do_package
is installed. If it isn't, its requiring gcc to rebuild.

If gcc-runtime do_package_write_xxx and do_packagedata is available, there
is no reason do_package should be needed.

The reason this is happening appears to be from the batching up of task
dependencies code, rather than setscene tasks stopping when passing over
a setscene task, they were being carried forward. This patch fixes it
so the data is 'zeroed' when passing over a setscene task boundary,
which gives the dependency graph that is expected.

After this patch, lsof will rebuild quite happily without
gcc-runtime:do_package being present, as expected. This should lead to
less dependencies being installed for builds from sstate and generally
better performance in general.

(Bitbake rev: f8bcb0a1e3b008b71c9a7cd21f76d0906f2d8068)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:26:33 +00:00
Ed Bartosh 2ff4ccb13f bitbake: buildinfohelper: add provides info to the db
Added new entries to Provides model and link them to
Recipe_Dependency using 'via' field.

This data will be used by Toaster UI to show 'Provides:'
information for the recipes.

[YOCTO #6169]

(Bitbake rev: 336ddc8df611d4c8f1c3d3a06d0a85bb544c38bc)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:26:32 +00:00
Ed Bartosh 6a28ed33b7 bitbake: buildinfohelper: use providermap
Used providermap in store_dependency_information function
to find virtual dependencies. This should fix annoying
warnings "stpd: KeyError saving recipe dependency"

[YOCTO #6169]

(Bitbake rev: 85c416ca338c886db6e79651e44727482df9fb07)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:26:32 +00:00
Ed Bartosh f2b7252190 bitbake: cooker: add providermap to dep_tree
Added providermap information to the result of buildDependTree API.

This will be used by Toaster to map virtual dependencies to recipes.

(Bitbake rev: d3e07368549f30265f59846a260efa8230a225ca)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:26:32 +00:00
Ed Bartosh 7e380d460c bitbake: taskdata: refactor get_providermap
Added optional parameter 'prefix' to filter out names that
don't start with specified prefix. Changed existing call
of get_providermap according to changed API.

Optimized the code: got rid of extra loop and temporary
list variable virts.

(Bitbake rev: df5a1392d6f91ccb44a99721c7d847da242121bb)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:26:32 +00:00
Richard Purdie 46731daa5f bitbake: main/runqueue: Add --setscene-only option to bitbake
Its turning out that we really need a way to have bitbake just run
the setscene tasks but not any real tasks, particularly for SDK
operations.

Add an option for this since its pretty straight forward. This allows
various nasty workarounds in OE-Core to be removed.

(Bitbake rev: e4a2aafa1650a227a04d92a8a0b31efaed2c310e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:26:32 +00:00
Richard Purdie 221d864042 bitbake: fetch/git: Change to use clearer ssh url syntax for broken servers
Some servers, e.g. bitbucket.org can't cope with ssh:// as part of
the git url syntax. git itself is happy enough with this but you
get server side errors when using it.

This changes the git fetcher to use the more common ssh url format
which also means we need a : before the path.

Seems a shame to have to do this due to broken servers however
it should be safe enough since this other form is the one most people
use on the commandline so it should be safe enough.

[YOCTO #8864]

(Bitbake rev: 4193e99adce8e88f12ac88d7578ad39575f7e346)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-08 13:58:08 +00:00
Richard Purdie 46d62d076d bitbake: knotty: Use non-interactive mode as fallback for dumb terminals
TERM=dumb bitbake X

shows no output for task status which is suboptimal. Use the non-interactive
mode if the terminal doesn't support what we need for interactive mode giving
a better user experience. Also print a note to the console to say this has
happened.

[YOCTO #8768]

(Bitbake rev: 6f84cf4bd77f35fcd07e0b2f5149f1d6866a414d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-08 13:58:08 +00:00
Ross Burton bfa7859ffa bitbake: cooker: fix findFilesMatchingInDir documentation
The documentation for findFilesMatchingInDir() was inconsistant with the
implementation: the regex was escaped before searching so effectively it's a
pure textual substring, and the machine example was broken.

(Bitbake rev: 6bef981488ec94b46dbe3797acfecf9c4b6ecbbc)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-08 13:58:08 +00:00
Ross Burton 3d4273716d bitbake: cooker: use in instead of count
No point counting all instances when we just want to know if there's any or not.

(Bitbake rev: 3369072efb653339da8dbd1ca864ff8e1ff899ca)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-08 13:58:08 +00:00
Richard Purdie f8185ff9f8 bitbake: ast: Add filename/lineno to mapped functions
Where we add in mappings for EXPORT_FUNCTIONS, add dummy filename
and lineno data so ensure the assumption that all python functions
have this is correct.

(Bitbake rev: 547128731e62b36d2271c4390b3fee2b16c535dc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:35 +00:00
Ed Bartosh a178c5a163 bitbake: main: kill server without queue setup
Setup of event queue includes registering of UI handler.
This operation can fail when cooker is busy. However, there is
no need in registering UI handler for terminating the server.

Moved the call of connection.terminateServer before setting up
of the event queue. This should make terminating server to work
more reliably as it doesn't depend on setting up the event queue
and registering UI handler anymore.

This should also help Toaster backend to restart bitbake server
and observer without getting "Could not register UI event handler"
errors.

[YOCTO #8776]

(Bitbake rev: 0c5a9349f797d05c282c2ada1893e187e05f0576)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:35 +00:00
Ed Bartosh 773700dbc3 bitbake: xmplrpc: split connect method
Current code in connect method sets up event queue, which requires
registering UI handler. This functionality may not be needed for
some operations, e.g. for server termination.

Moved functionality of setting up event queue in from 'connect'
method to 'setupEventQueue' in BitBakeXMLRPCServerConnection class.

(Bitbake rev: 4429871da76d6bd29e023ff42740fe7daa6b40fa)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:35 +00:00
Ed Bartosh 05b4fbc947 bitbake: uievent: refactor retry loop
Replaced 'while' loop with 'for' loop.
Made the code more compact and hopefully more understandable.

(Bitbake rev: 4e1e497c8432536b3522295e5b1284844ccea056)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:35 +00:00
Ed Bartosh ebc169c360 bitbake: uievent: get rid of EventHandler attribute
This attribute was introduced by mistake. EventHandle is used in the
code for the same purpose.

(Bitbake rev: 8d505ec8913a7d51de48b4f52bb64c5d6a0bb08e)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:35 +00:00
Ed Bartosh 4e0de6eca2 bitbake: uievent: add error to registerEventHandler return
Current code throws Exception("Could not register UI event handler")
if event handler can't be registered. The real reason of this is that
cooker is in busy state. Error message lacks information about this.

Added error message to the return value of registerEventHandler.
Included returned error message into the log message and exception
text.

(Bitbake rev: 07de1ca7d57dcd0cc37406feae2949da12a3fa7a)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:34 +00:00
Ed Bartosh 01419d5565 bitbake: cooker: add state.get_name method
Added method to convert state code into the human readable name.
It will be used in logging and error reporting.

(Bitbake rev: 9ec6379b27d210214d0b3f2e55962f721b7f5f51)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:34 +00:00
Juro Bystricky 763506d49b bitbake: fetch2/__init__.py: Add support for 7-Zip
7-Zip is a file archiver claiming the highest compression ratio.
This patch allows using 7-Zip commpressed files in bitbake recipes.
Two common formats are supported:

SRC_URI = "file://abc.tar.7z"
SRC_URI = "file://abc.7z"

(Bitbake rev: 7120f5bfaae54e91bc95da5667831424724ce613)

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:34 +00:00
Richard Purdie f5bfc1cc26 bitbake: utils: Remove double compile from better_compile
Poking around the ast to correct linenumbers works well for runtime failures
but not for parsing ones. We can use blank linefeeds to correct the line
numbers instead, with the advantage that we don't need to double compile.

(Bitbake rev: 10256ac3e7be7e691176ecc5d55856d88f1fe940)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:34 +00:00
Robert Yang b4141f6494 bitbake: fetch2/local.py: avoid using PREMIRROR
The PREMIRROR isn't useful for "file://", so avoid using it, this is
good for searching speed and can reduce useless lines in log.do_fetch.

(Bitbake rev: e130dca85bac82bd4d88f94a6bf9fe36e8ad4d7c)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:34 +00:00
Richard Purdie 1ad3595e8c bitbake: siggen: Change exception note into a warning
This makes no sense as just a note, its at least a warning and useful
to get an idea of which codepath is failing.

(Bitbake rev: 0194cf0da24dc72dab0612cd54aa5190e6cd92f2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:34 +00:00
Richard Purdie 4ba49aca42 bitbake: data: Drop misleading ExpansionError exception
This codepath can be triggered by a python indentation error for example.
Showing it as an ExpansionError is misleading.

Change the code to add a warning about where the failure came from (in
particular giving the variable key name that triggered it) but raise the
proper exception.

(Bitbake rev: d49d46533704e8b4404e29abfb5a7383d704c91a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:34 +00:00
Richard Purdie 2c9431135d bitbake: cooker: Drop useless parsing exception
The SyntaxError exception simply shows the recipe that failed to parse
which is pretty useless without the actual exception. We could make it
print more info, however we can just use one of the more generic handlers
instead and remove this one.

For a python indentation error, this leads to a much more readable error
message.

(Bitbake rev: 9241eb10847634e34c5ff8767ed8c114f66ff6cf)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:34 +00:00
Richard Purdie a16b543436 bitbake: data: Pass lineno/filename data from build_dependencies
If parse_python() fails, the output is confusing. Passing in the extra
file/line data isn't expensive and improves readability significantly.

(Bitbake rev: a4bb753488d322e0e31c31d6377ba780f2f824c4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:34 +00:00
Richard Purdie 958f0ff56b bitbake: codeparser: Add support for correct linenumbers
Currently, if there is something like a python indentation error in a
python function, the linenumbers and even file aren't reported correctly.

This allows lineno and filename parameters to be passed in to correct this.

The lack of a lineno parameter to python's compile() function is worked
around by using empty linefeeds. Ugly, but effective and with minimal
performance overhead.

(Bitbake rev: 5796ed550d127853808f38257f8dcc8c1cf59342)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:34 +00:00
Richard Purdie 0381b78aa4 bitbake: event/utils/methodpool: Add a cache of compiled code objects
With the addition of function line number handling, the overhead of
the compile functions is no longer negligible. We tend to compile
the same pieces of code over and over again so wrapping a cache around
this is beneficial and removes the overhead of line numbered functions.

Life cycle of a cache using a global like this is in theory problematic
although in reality unlikely to be an issue. It can be dealt with
if/as/when we deal with the other global caches.

(Bitbake rev: 98d7002d1dca4b62042e1589fd5b9b3805d57f7a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-22 00:02:05 +00:00
Richard Purdie c61c1eb26a bitbake: BBHandler: Improve IN_PYTHON_EOF handling
Now we're actively using the line numbers for other thins, having
magic values like IN_PYTHON_EOF causes problems, in particular, 32
bit overflow on 32 bit machines.

There is a neater way to signal eof to feeder(), just using an extra
parameter so use this instead and drop the IN_PYTHON_EOF magic values.

This has the added bonus that line numbers are then correct for
python functions at the end of files.

(Bitbake rev: e0f05871c2a6f1e86ae19ad343c7c6f822ddb67e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-22 00:02:05 +00:00
Richard Purdie 522dcaa0cb bitbake: knotty: Improve exception error message
Instead of:
"""
can only concatenate tuple (not "int") to tuple
"""
we now see:

"""
Traceback (most recent call last):
  File "/media/build1/poky/bitbake/lib/bb/ui/knotty.py", line 324, in main
    termfilter.updateFooter()
  File "/media/build1/poky/bitbake/lib/bb/ui/knotty.py", line 210, in updateFooter
    lines = 1 + int(len(content) / (self.columns + 1))
TypeError: can only concatenate tuple (not "int") to tuple
"""

which makes tacking down and fixing the problem much easier.

Also ensure we set an error exit code.

(Bitbake rev: d965bcae6cfd268406a3bd1ef77c5bb6c6e1c6d7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 12:18:19 +00:00
Richard Purdie 01d67bfa77 bitbake: knotty: Fix row/column function return value issue
When the row handling was introduced, one of the callbacks was
missed resulting in:

TypeError: can only concatenate tuple (not "int") to tuple

Fix it.

(Bitbake rev: 0b77cea2bf5b5f5704e2650fb0332f5d78037781)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 12:18:19 +00:00
Richard Purdie 6c12efa5c2 bitbake: buildinfohelper: Update for buildstats layout change
This updates buildinfo helper for the recent buildstats layout change

(Bitbake rev: 30311bbe667e9f22de17fae00ff58da06a7c3e23)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 12:18:19 +00:00
Christopher Larson 28ea1a1c95 bitbake: fetch: use orig localpath when calling orig method
When a mirror tarball is fetched, the original fetch method is called, which
unpacks the mirror tarball. After the original method is called, it checks the
localpath of the mirror tarball rather than the clone path, which isn't ideal,
particularly if the mirror tarball was removed due to being out of date. We
know the original fetch method will do what it needs to do to get its content
in the form it needs from the mirror tarball, so we can use its localpath
instead.

(Bitbake rev: 1732ad65d6c7d67b7d07cb30c074f5016adadbea)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Awais Belal <awais_belal@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 12:18:19 +00:00
Richard Purdie 5cb6d83f17 bitbake: utils: Improve traceback from better_exec internal errors
If you break the internals of better_exec(), you get a very weird
error about tb_next not being a method of None. Fix this by checking
we can step back a trace level.

(Bitbake rev: 1d710ed484f68fca0789022dde7ba877b9a894f5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 12:18:19 +00:00
Richard Purdie 0019edc818 bitbake: ast/event/utils: Improve tracebacks to include file and line numbers more correctly
Currently bitbake tracebacks can have places where the line numbers are
inaccurate and filenames may be missing. These changes start to try and
correct this.

The only way I could find to correct line numbers was to compile as a
python ast, tweak the line numbers then compile to bytecode. I'm open
to better ways of doing this if anyone knows of any.

This does mean passing a few more parameters into functions, and putting
more data into the data store about functions (i.e. their filenames
and line numbers) but the improvement in debugging is more than worthwhile).

Before:
----------------
ERROR: Execution of event handler 'run_buildstats' failed
Traceback (most recent call last):
  File "run_buildstats(e)", line 43, in run_buildstats(e=<bb.build.TaskStarted object at 0x7f7b7c57a590>)
NameError: global name 'notexist' is not defined

ERROR: Build of do_patch failed
ERROR: Traceback (most recent call last):
  File "/media/build1/poky/bitbake/lib/bb/build.py", line 560, in exec_task
    return _exec_task(fn, task, d, quieterr)
  File "/media/build1/poky/bitbake/lib/bb/build.py", line 497, in _exec_task
    event.fire(TaskStarted(task, logfn, flags, localdata), localdata)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 170, in fire
    fire_class_handlers(event, d)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 109, in fire_class_handlers
    execute_handler(name, handler, event, d)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 81, in execute_handler
    ret = handler(event)
  File "run_buildstats(e)", line 43, in run_buildstats
NameError: global name 'notexist' is not defined

----------------

After:
----------------
ERROR: Execution of event handler 'run_buildstats' failed
Traceback (most recent call last):
  File "/media/build1/poky/meta/classes/buildstats.bbclass", line 143, in run_buildstats(e=<bb.build.TaskStarted object at 0x7efe89284e10>):
         if isinstance(e, bb.build.TaskStarted):
    >        trigger = notexist
             pn = d.getVar("PN", True)
NameError: global name 'notexist' is not defined

ERROR: Build of do_package failed
ERROR: Traceback (most recent call last):
  File "/media/build1/poky/bitbake/lib/bb/build.py", line 560, in exec_task
    return _exec_task(fn, task, d, quieterr)
  File "/media/build1/poky/bitbake/lib/bb/build.py", line 497, in _exec_task
    event.fire(TaskStarted(task, logfn, flags, localdata), localdata)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 170, in fire
    fire_class_handlers(event, d)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 109, in fire_class_handlers
    execute_handler(name, handler, event, d)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 81, in execute_handler
    ret = handler(event)
  File "/media/build1/poky/meta/classes/buildstats.bbclass", line 143, in run_buildstats
    trigger = notexist
NameError: global name 'notexist' is not defined
----------------

(Bitbake rev: 1ff860960919ff6f8097138bc68de85bcb5f88b0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 12:18:19 +00:00
Richard Purdie b14ccb2367 bitbake: runqueue: Add support for <task>- syntax
It can be useful to run all tasks up to but not including a specific task. The
main reason this was never added was the lack of a good syntax. This patch
uses the syntax <taskname>- to denote this behaviour which is simple, not
invasive and fits what we need from good syntax IMO, hence we can add this.

(Bitbake rev: 99ccfd411ab3f7baa111f9f3d50fae68816a9a83)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 12:18:19 +00:00
Elliot Smith 23c48065d2 bitbake: toaster: Start Django machinery for database access
buildinfohelper, with the new import paths for our Django models
and Django 1.8, was not getting an active connection to the database.

In buildinfohelper, call django.setup() explicitly to make sure
that the database connection is ready and models can be queried
and saved.

[YOCTO #8364]

(Bitbake rev: 671aaab8cb7c494cd5c7621b45a6f41a203d8bb5)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:06 +00:00
Elliot Smith 7a0c45e478 bitbake: toaster: Create default project with get_or_create* method
Rather than maintain data as part of the migrations (as was
done for the default project previously), create the default
(cli builds) project on demand as a by-product of getting
it from the database.

[YOCTO #8364]

(Bitbake rev: 5fd8e90ab9b81d1bd0d301bc1c91228ecbbea74b)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:06 +00:00
Ed Bartosh 9de8dfa11a bitbake: toaster: Fix references to app paths
The buildinfohelper runs outside of Django, but still needs
access to our Django app classes.

Previously, the imports referenced the toaster.* app, which worked
fine. But in Django 1.8, this causes an error about the same
module being loaded multiple times from different paths.

Change the paths to our Django modules so they don't cause
this error to be thrown. We can do this as we've added our
application libraries to sys.path in the buildinfohelper anyway.

[YOCTO #8364]

(Bitbake rev: 070da64cf32c32b5ffc34d611b463c3a3960b419)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:06 +00:00
Ed Bartosh b322dec016 bitbake: toasterui: process SetBRBE event
Currently BRBE is known to toasterui only when build is
started. It's passed to it with BuildStarted event. This is
too late as if build fails earilier than build starts toasterui
can not inform Toaster about the failure.

Set BRBE as soon as it's provided by Toaster.
This should make toasterui to be able to inform Toaster
about early build failures, e.g. failures during recipe parsing.

(Bitbake rev: d7819508dac488a64be3caec88db285cda9599ab)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:06 +00:00
Richard Purdie 5de3800c30 bitbake: event: Fix subprocess event error traceback failures
If subprocess raises a CalledProcessError() error, e.g. from a call
like subprocess.check_call("false"), bitbake would try and pass the
object over IPC and fail, leading to an unusual error:

('__init__() takes at least 3 arguments (1 given)', <class 'subprocess.CalledProcessError'>, ())%

To avoid this, we turn the value into a string which prevents the
issues the IPC has trying to deal with the object (for the same reason
we deal with tracebacks here too).

[YOCTO #8752]

(Bitbake rev: 05695424b918fc81b16cbac70d79d8271a0b6045)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:06 +00:00
brian avery 7e8ff7b9d7 bitbake: toaster: toasterui Add ParseStarted/ParseProgress events to mask
Toaster is not able to see ParseStarted and ParseProgress events
for command-line builds. This means it's not possible for Toaster
to detect failed builds, if the failure occurs at a point before
the BuildStarted event, as the build won't show up at all.

Add these events to the event mask, so that Toaster's toasterui
can detect and respond to them.

(Bitbake rev: 16bfd3e3d145705a2b3a05648ddbcacc7a338dfa)

Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-09 09:03:35 +00:00
Richard Purdie f9384b0669 bitbake: knotty: Enforce terminal line limit to stop crazy scrolling
If there are more tasks running than there are lines on the terminal, the data
scrolls in ways the UI wasn't designed for. This patch adjusts the UI just to
show the processes which fit onto the number of rows in the terminal window.
You can see the total number running from the counter in the top left as usual
and this makes warning and errors messages scrolling from the top of the window
work as designed.

Ultimately, scrolling would be nice but is for another time, this fixes the
biggest UI issue on highly parallel machines.

(Bitbake rev: 67b77658e2bfa849f6f55c9c262cb11d6bfdb399)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-08 10:31:48 +00:00
Alexander Kanavin 3745479596 bitbake: bitbake: rename REGEX, REGEX_URI, and GITTAGREGEX.
Rename REGEX to UPSTREAM_CHECK_REGEX, REGEX_URI to UPSTREAM_CHECK_URI, and
GITTAGREGEX to UPSTREAM_CHECK_GITTAGREGEX to better reflect their purpose
and to reflect a common namespace.

(Bitbake rev: f0a9e783f9969573fd74edfa241ef14f18ac684e)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-08 10:20:31 +00:00
Ed Bartosh 8b7a548e30 bitbake: toaster: get rid of complicated heuristics
Removed buildinfohelper code which was trying to guess layer version of
the recipe using build request information. The code caused creation of
duplicated recipes as it resulted in layer version from layer index
instead of returning build layer version. As a result of this Toaster
UI was not showing any information about recipes.

Default approach used to find layer version seems to work much better as
it finds proper layer version. Now toaster will use it as the only
way to find layer version.

(Bitbake rev: 101690bda7ad55dc0657483233c90c374713755b)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:11 +00:00
Michael Wood 53e981e583 bitbake: toaster: buildinfohelper Broaden the toaster created recipe data case
When build information is the 'original' source of the information we
need to return the recipe that was created rather than the copy of the
recipe that is taken for keeping build history. We do this already for
command line triggered builds, but we also have this case for custom
images. We can simply check if the built_recipe exists instead of
special casing this.

(Bitbake rev: 9a8653bf602b2111dee7ee6a459682a68a695b22)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:10 +00:00
Ed Bartosh 57e5f24e57 bitbake: toaster: do not create duplicate HelpText objects
buildinfohelper code expects only one HelpText object per
build/variable/description.
Current code creates more than one such an object, which causes
toastergui to crash with this exception:
 MultipleObjectsReturned: get() returned more than one HelpText -- it returned 2!

Used git_or_create API to ensure that only one HelpText object is
created.

(Bitbake rev: e9b46803eb6f1f4044919abf90c8aeb3536e73ed)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-07 17:37:10 +00:00