Commit Graph

3240 Commits

Author SHA1 Message Date
Paul Eggleton fac16ff8f7 bitbake: siggen: properly close files rather than opening them inline
If you don't do this, with Python 3 you get a warning on exit under some
circumstances.

(Bitbake rev: 49502685df3e616023df352823156381b1f79cd3)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18 10:06:26 +01:00
Jérémy Rosen 0eb6d709b6 bitbake: ast/ConfHandler: Add a syntax to clear variable
unset VAR
will clear variable VAR
unset VAR[flag]
will clear flag "flag" from var VAR

(Bitbake rev: bedbd46ece8d1285b5cd2ea07dc64b4875b479aa)

Signed-off-by: Jérémy Rosen <jeremy.rosen@openwide.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18 10:06:26 +01:00
Richard Purdie b50b14e372 bitbake: cache: Build datastores from databuilder object
Rather than passing in a datastore to build on top of, use the data builder
object in the cache and base the parsed recipe from this. This turns
things into proper objects building from one another rather than messy
mixes of static and class functions.

This sets things up so we can support parsing and building multiple
configurations.

(Bitbake rev: fef18b445c0cb6b266cd939b9c78d7cbce38663f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18 10:06:26 +01:00
Richard Purdie b176189df1 bitbake: cache: Split Cache() into a NoCache() parent object
There are some cases we want to parse recipes without any cache
setup or involvement. Split out the standalone functions into
a NoCache variant which the Cache is based upon, setting the scene
for further cleanup and restructuring.

(Bitbake rev: 120b64ea6a0c0ecae7af0fd15d989934fa4f1c36)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18 10:06:26 +01:00
Richard Purdie e79550ea87 bitbake: cache/cooker: Pass databuilder into bb.cache.Cache()
Rather that the current mix of static and class methods, refactor
so that the cache has the databuilder object internally. This becomes
useful for the following patches for multi config support.

It effectively completes some of the object oriented work we've been
working towards in the bitbake core for a while.

(Bitbake rev: 7da062956bf40c1b9ac1aaee222a13f40bba9b19)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18 10:06:26 +01:00
Richard Purdie 97ce9126a6 bitbake: cache: Make virtualfn2realfn/realfn2virtual standalone functions
Needing to access these static methods through a class doesn't
make sense. Move these to become module level standalone functions.

(Bitbake rev: 6d06e93c6a2204af6d2cf747a4610bd0eeb9f202)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18 10:06:26 +01:00
Richard Purdie 4cd5647f12 bitbake: cache/ast: Move __VARIANTS handling to parse cache function
Simple refactoring to allow for multiconfig support.

(Bitbake rev: 266b848da40904446eb1d084bbdc5307a9b45197)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18 10:06:26 +01:00
Richard Purdie 0ef16f083e bitbake: runqueue: Abstract worker functionality to an object/array
With the introduction of multi-config and the possibility of distributed
builds we need arrays of workers rather than the existing two.

This refactors the code to have a dict() of workers and a dict of
fakeworkers, represented by objects. The code can iterate over these.

This is separated out from the multi-config changes since its separable
and clearer this way.

(Bitbake rev: 8181d96e0a4df0aa47287669681116fa65bcae16)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18 10:06:26 +01:00
Richard Purdie 249686927b bitbake: cookerdata: Simplify prefiles/postfiles
The current codepaths are rather confusing. Stop passing these
as parameters and use the ones from when the object is created.

(Bitbake rev: 8c992c148d9619b10eeae8bbd9376ecf408037a5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18 10:06:26 +01:00
Francisco Pedraza 500ebdda6b bitbake: bb/utils.py: export_proxies add GIT_PROXY_COMMAND
This was added to enable the usage of git through proxies.

(Bitbake rev: 449fc52e483a3bf1cec1c5d8cf8c3946ec5292ab)

Signed-off-by: Francisco Pedraza <francisco.j.pedraza.gonzalez@intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-17 10:22:58 +01:00
Paul Eggleton 8a45291164 bitbake: knotty: don't show number of running tasks in quiet mode
There's not a whole lot of point showing how many tasks are running when
we're in quiet mode, it just looks a bit strange particularly when it's
not running any tasks.

(Bitbake rev: 5317200d9cd73c6f971bc1b0cfe8692749e27e3a)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-17 10:22:58 +01:00
Paul Eggleton ea0800049d bitbake: knotty: fix task progress bar not starting at 0%
If we have the task number here we need to subtract 1 to get the number
of tasks completed.

(Bitbake rev: 7c78a1cd3f0638ae76f7c7a469b7f667c7c58090)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-17 10:22:58 +01:00
Paul Eggleton 1b6f701cd9 bitbake: runqueue: fix two minor issues with the initialising tasks progress
A couple of fixes for the "Initialising tasks" progress bar behaviour:
* Properly finish the progress bar when using bitbake -S
* Finish the progress bar before calling BB_HASHCHECK_FUNCTION (so that
  in OE when that shows its own "Checking sstate mirror object
  availability"  progress bar it gets shown on the next line as it
  should).

(Bitbake rev: de6759d8e9990e426e6d6464a2e05381cd4c12d6)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-17 10:22:58 +01:00
Ross Burton 407ba77fe2 bitbake: lib/bb/tests/fetch: remove URL that doesn't exist anymore
The CUPS ipptool URL we were checking now redirects to github where the tarball
isn't present, so remove it from the test suite.

(Bitbake rev: 4b50895fb3462b21e3874a2e99c363c8d05e89e6)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-17 10:22:58 +01:00
Michael Wood e274a5167e bitbake: toaster: buildinfohelper Add handling local layers (i.e. non-git) layers
Adds handling of the non-git layers to create and update the
corresponding layer objects in Toaster.

(Bitbake rev: 0a9b5d7d9655dbb09d458fc6e330e932f0f9dab6)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-11 00:09:30 +01:00
Elliot Smith f6261da9c0 bitbake: toasterui: ensure that the Build object is always available
Many of the methods in toasterui and buildinfohelper rely
on the internal state of the buildinfohelper; in particular, they
need a Build object to have been created on the buildinfohelper.

If the creation of this Build object is tied to an event which
may or may not occur, there's no guarantee that it will exist.
This then causes assertion errors in those methods.

To prevent this from happening, add an _ensure_build() method
to buildinfohelper. This ensures that a minimal Build object
is always available whenever it is needed, either by retrieving
it from the BuildRequest or creating it; it also ensures that
the Build object is up to date with whatever data is available
on the bitbake server (DISTRO, MACHINE etc.).

This method is then called by any other method which relies on
a Build object being in the internal state, ensuring that the
object is either available, or creating it.

(Bitbake rev: 0990b4c73f194ec0be1762e4e48b1a525d8349fb)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-11 00:09:27 +01:00
Elliot Smith dd99cf957d bitbake: toaster: show progress of recipe parsing in recent builds area
Modify buildinfohelper and toasterui so that they record the
recipe parse progress (from ParseProgress events in bitbake)
on the Build object.

Note that because the Build object is now created at the
point when ParseStarted occurs, it is necessary to set the
build name to the empty string initially (hence the migration).
The build name can be set when the build properly starts,
i.e. at the BuildStarted event.

Then use this additional data to determine whether a Build
is in a "Parsing" state, and report this in the JSON API.
This enables the most recent builds area to show the recipe
parse progress.

Add additional logic to update the progress bar if the progress
for a build object changes.

[YOCTO #9631]

(Bitbake rev: f33d51d46d70e73e04e325807c1bc4eb68462f7b)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-11 00:09:26 +01:00
Elliot Smith c471740f5b bitbake: cooker: add BuildInit event
In situations where a bitbake run fails before the build
properly starts and BuildStarted is fired, a UI has no way
to get at the targets passed to the build. This makes it
difficult for the UI to report on the targets which failed.

Fire a BuildInit event before running buildTargets() or
buildFile(). This enables a UI to capture targets passed to
buildTargets(), even if the build fails (e.g. the targets
themselves are invalid).

[YOCTO #8440]

(Bitbake rev: ac02fda870965bf7d44ff5688eda54d2d11ab9c7)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-11 00:09:26 +01:00
Ulrich Ölmann 5d1d1774ec bitbake: siggen: Fix typo
(Bitbake rev: 40f605199fb3ec2549611508b7576c64d735b2b7)

Signed-off-by: Ulrich Ölmann <u.oelmann@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-11 00:07:09 +01:00
Terry Boese 93817932de bitbake: fetch2/gitannex.py: use 'git annex init' instead of 'git annex sync'
The git annex fetcher needs git annex to be initialized.  Previously
it was using 'git annex sync' to do this, but that has the downside
of moving the checkout to the tip of the default branch.  This means
that tags, SRCREV, etc don't work in the gitannex case.

(Bitbake rev: c1a57e2dd7fc96834643be5591a96f239215481a)

Signed-off-by: Terry Boese <terry.boese@vecima.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-11 00:07:08 +01:00
Richard Purdie 82da1e6f6c bitbake: persist_data: Fix py3 update stack overflow
Revision d0f904d407f57998419bd9c305ce53e5eaa36b24 accidentally broke
items() and values() and made them cause stack overflows. Undo that
breakage.

(Bitbake rev: 88c5beca705efa7df4a96fb2aaf3f13c336ac328)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-11 00:07:08 +01:00
Mariano Lopez a6630c2310 bitbake: fetch: Fix use of tar's --exclude option for tar >= 1.29
Starting from tar 1.29 the --exclude option won't work
anymore if is not used before the path. There are some
fetch modules that copy the ptest using tar and --exclude
option. This fixes these for bitbake.

[YOCTO #9763]

(Bitbake rev: cc71d5d9da71ea5f21d02f3b2fbf119bd2d794f0)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-10 10:46:34 +01:00
Paul Eggleton 2a849e7e9b bitbake: lib/bb/checksum: avoid exception on broken symlinks
If using OE's externalsrc with a source tree that is not tracked by git
and contains broken symlinks, you can receive "TypeError: unorderable
types: NoneType() < str()" within the file checksum code due to:

 checksums.sort(key=operator.itemgetter(1))

Don't add files with no checksum to the checksums list in order to avoid
this.

(Bitbake rev: 484fe5a3f5b840e5422cbdff0eef9aecfe944a19)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-29 09:53:32 +01:00
Ross Burton ea234239f4 bitbake: lib/bb/utils: show subprocess output in stack traces
If better_exec() throws a subprocess.CalledProcessError then show the output to
the user as it likely contains useful information for solving the problem.

(Bitbake rev: 8a6424ed871c3cbacd21cae8bc801197f83d67a6)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-29 09:53:32 +01:00
Maxin B. John c42b5333f0 bitbake: fetch2: fix pickle issues while switching from master to krogoth
While switching from master to krogoth build with a common download directory,
got a large number of warnings like the one listed below:

WARNING: freetype-2.6.3-r0 do_fetch: Couldn't load checksums from
donestamp /home/maxin/downloads/freetype-2.6.3.tar.bz2.done: ValueError
(msg: unsupported pickle protocol: 4)

These warnings are caused by the difference in pickle module
implementation in python3(master) and python2(krogoth). Python2 supports
3 different protocols (0, 1, 2) and pickle.HIGHEST_PROTOCOL is 2 where as
Python3 supports 5 different protocols (0, 1, 2, 3, 4) and
pickle.HIGHEST_PROTOCOL is obviously 4.

My suggestion is to use 2 since it is backward compatible with python2
(all the supported distros for krogoth provides python2 which supports
pickle protocol version 2)

(Bitbake rev: cc67800f279fb211ee3bb4ea7009fdbb82973b02)

Signed-off-by: Maxin B. John <maxin.john@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-29 09:53:32 +01:00
Ross Burton 1393b23f51 bitbake: lib/bb/build: handle incomplete message fragments in log FIFO
It's possible that the logging FIFO doesn't do a complete read (or the sender a
complete write) with the result that an incomplete message is read in bitbake.
This used to result in silently truncated lines but since 42d727 now also
results in a warning as the start of the rest of the message isn't a valid
logging command.

Solve this by storing incoming bytes in a bytearray() across reads, and parsing
complete messages from that.

[ YOCTO #9999 ]

(Bitbake rev: 508112793ee7ace613f07695222997309a2ca58f)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:37 +01:00
Mario Domenech Goulart 24905d3c2d bitbake: fetcher2/__init__: Print command in case of ExecutionError in runfetchcmd
(Bitbake rev: df7f4897c463a48c45514e2bcbd44cc7f86c4bb0)

Signed-off-by: Mario Domenech Goulart <mario.goulart@bmw-carit.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:36 +01:00
Michael Wood 97278fb51c bitbake: toaster: orm Remove the layerindex specific up_branch fields
We don't need to keep track of layerindex data in our database. And
using branch==release is very confusing in the schema. Instead use the
existing Release definition to keep track of which release a
layer_version is for.

Remove the Branch model and all references to it.

Create a migration path to convert from up_branches to their
corresponding releases.

(Bitbake rev: f8f4cffe6fd371f3a7e63690c68f3fcb5dc1f297)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:36 +01:00
Paul Eggleton c27ae255db bitbake: lib/bb/progress: avoid possibility of start event being reported twice
In MultiStageProgressReporter, set a guard when we start the progress
so that it can't happen more than once. This fixes "Initialising
tasks.." being shown twice in succession when running bitbake in
non-interactive terminal mode.

(Bitbake rev: 923e68e069127ee7f6e11b91eb1cfa09d502a110)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:35 +01:00
Paul Eggleton 5f4559b2eb bitbake: knotty: don't display ETA for tasks with progress
It turns out that progress information we can extract from a task is
rarely apportioned closely enough to the time taken for the ETA to be
accurate, so showing it is going to be misleading most of the time for
anything but the most basic of examples. Let's just remove it and avoid
misleading (or worse, annoying) the user.

Fixes [YOCTO #9986].

(Bitbake rev: 235db4870b11db97250979e647b54cdb5ce4fbb6)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:35 +01:00
Paul Eggleton 23a551bea1 bitbake: knotty: fix some minor bugs in BBProgress
If you specify custom widgets then we don't want to assume where the
"extra" position is - you should have to specify it, and if it isn't
specified it shouldn't just wipe out the last widget or you can start to
see odd behaviour if you're modifying the code.

(Bitbake rev: 19e33c10feb1637589ceb05b5e8d58b1e012ccb8)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:35 +01:00
Enrico Scholz 0c3ce68410 bitbake: fetch: copy files with -H
When using a PREMIRROR with plain (non-unpack) files, a SRC_URI like

SRC_URI = "file://devmem2.c"

will cause devmem2.c to be a symlink in the WORKDIR pointing to the
local PREMIRROR.

Trying to apply a patch on this file will either modify the file on
the PREMIRROR or will fail due to sanity checks:

ERROR: devmem2-1.0-r7 do_patch: Command Error: 'quilt --quiltrc /cache/build-ubuntu/sysroots/x86_64-oe-linux/etc/quiltrc push' exited with 1  Output:
Applying patch devmem2-fixups-2.patch
File devmem2.c is not a regular file -- refusing to patch

(Bitbake rev: cfd481fe9799e7a4c6bfac32e56cc91cfcd81088)

Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:35 +01:00
Richard Purdie 7f6b6b2ab9 bitbake: cache: Don't interleave pickle cache file writing
For some reason the data written in this way is coming back out the
files out of order. I've not been able to simplify the test case to a
point where this was standalone reproducible. Simplify the code and
write out the cache files sequentially since this seems to avoid the
errors and makes the code more readable.

(Bitbake rev: 14ec47f5f0566dbd280fae8a03160c8500ad3929)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:35 +01:00
Richard Purdie 5d41200113 bitbake: cache: Add better cache loading sanity checks
We've seen cache corruption where the pairs come out in a different
order to the way we saved them for unknown reasons. Add better sanity
checking to give a more user friendly error rather than a crash/traceback.

Also allows the system to reparse and recover.

(Bitbake rev: 4be4a15491530bd6dc018033ad3d4b2562ab6e23)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:35 +01:00
Richard Purdie 40d45cf7a6 bitbake: cache: Drop/simplify pointless type checking
Since we no longer have random data like version fields in these structures
and we can assume any extra cache data subclasses our class, simplify the
code.

This is mostly reindenting after removal of the pointless type checks.

(Bitbake rev: 5eb36278ac9975de1945f6da8161187320d90ba7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:35 +01:00
Richard Purdie c9e65c5d29 bitbake: cache: Improve versions fields handling
Firstly, don't store the versions fields in memory in the cache objects
data store. This just complicates the code for no good reason.

Secondly, write the version fields to all cache files, not just the
core one. This makes everything consistent and easier.

(Bitbake rev: cb666262b2f986b5d9331dfb30458ef1a151fa4d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:35 +01:00
Richard Purdie 09f6a56aa4 bitbake: cache: Correctly handle missing extra caches
If an "extras" cache file is corrupted, the system would not notice
and later fail with errors about missing entries. Add a test for this
which means we can fall back to re-parsing in those cases.

[YOCTO #9902]

(Bitbake rev: 51843d8f2bbe2e54db7593ca61984abe70423ef6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:34 +01:00
Richard Purdie b2ef908be1 bitbake: cache: Move the parsing message to a more logical place
Otherwise you can look at the log and wonder why parsing isn't happening
when it really is due to other code paths.

(Bitbake rev: b48d95677a4d285a77cda2892179965f7f8f06dd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:34 +01:00
Ed Bartosh 317708235b bitbake: bitbake: implement idle timeout for xmlrpc server
Idle timeout can be specified either by -T/--idle-timeout option or
by sessing BBTIMEOUT environment variable. Bitbake xmlrpc server
will unload itself when timeout exprired, i.e. when server is idle
for more than <idle timeout> seconds.

[YOCTO #5534]

(Bitbake rev: 5fa0b3a19e7d0f40790f737abeddf499d53f1f6a)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-21 07:48:52 +01:00
Ed Bartosh 4cba010529 bitbake: bitbake: implement --foreground command line option
This option makes bitbake xmlrpc server to run in foreground.
It should be useful for debugging purposes.

(Bitbake rev: 9d4254be5853a546a346bf0d19919dcfba12773d)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-21 07:48:52 +01:00
Richard Purdie 90436919f8 bitbake: cooker: Fix incorrect dot file generation
In the runqueue cleanup/conversion, "dep" was mistakenly used where "tid" should
be leading to incorrect task-depends.dot files and causing general confusion.
Fix this, its clearly incorrect looking at the code.

(Bitbake rev: 689730dbb068c5ea3593e7b92fe5d5e5c0c3760a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-20 16:46:12 +01:00
Ed Bartosh 63ff759627 bitbake: bitbake: main: implement server autostart feature
If environment variable BBSERVER == 'autostart' bitbake will
automatically load server if it's not running yet.

If host and port are in bitbake.lock then bitbake tries to check
if server is running and responses to commands and starts new
server only if this check fails.

[YOCTO #5534]

(Bitbake rev: 89c6e625d47303b2aad8e6645762f17aee01b2d4)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-20 10:23:46 +01:00
Ed Bartosh b190c08b48 bitbake: bitbake: utils: add BBSERVER to the list of preserved variables
All environment variables that are not in the list returned by
preserved_envvars_exported are cleaned by bb.utils.clean_environment.

Added BBSERVER to the list as we need to access it in bb/main.py
after the call of bb.utils.clean_environment.

(Bitbake rev: 15c4ea679f4fe097a9f21cccfc82907b5f39a4e4)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-20 10:23:46 +01:00
Ed Bartosh 6597ba6b43 bitbake: bitbake: xmlrpc: implement check of connection to server
Implemented check_connection function. The purpose of this function
is to check if bitbake server is accessible and functional.
To check this this function tries to connect to bitbake server and
run getVariable command.

This API is going to be used to implement autoloading of bitbake
server.

[YOCTO #5534]

(Bitbake rev: 1a18f5ceb478f766b53850451549333f655621ea)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-20 10:23:45 +01:00
Ross Burton dead7b22b4 bitbake: lib/bb/utils: no need to unsetenv when manipulating os.environ
Doing both os.unsetenv(foo) and then del os.environ[foo] is pointless as del
will call unsetenv automatically.

(Bitbake rev: a4463e2ff3c7d234320176d671719243292f1af0)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-20 10:23:45 +01:00
Paul Eggleton 45baa90c8c bitbake: fetch2: implement progress support
Implement progress reporting support specifically for the fetchers. For
fetch tasks we don't necessarily know which fetcher will be used (we
might initially be fetching a git:// URI, but if we instead download a
mirror tarball we may fetch that over http using wget). These programs
also have different abilities as far as reporting progress goes (e.g.
wget gives us percentage complete and rate, git gives this some of the
time depending on what stage it's at). Additionally we filter out the
progress output before it makes it to the logs, in order to prevent the
logs filling up with junk.

At the moment this is only implemented for the wget and git fetchers
since they are the most commonly used (and svn doesn't seem to support
any kind of progress output, at least not without doing a relatively
expensive remote file listing first).

Line changes such as the ones you get in git's output as it progresses
don't make it to the log files, you only get the final state of the line
so the logs aren't filled with progress information that's useless after
the fact.

Part of the implementation for [YOCTO #5383].

(Bitbake rev: 4027649f422ee64b1c4e1ad8d48ac295050afbff)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 15:04:37 +01:00
Elliot Smith 2db40e3771 bitbake: toaster: add package manifest path to Target objects
Store the path to the *.rootfs.manifest file for targets which
generate images.

A link to the package manifest is displayed in the build dashboard
for targets which produce image files.

Like the license manifest path, if a target would have produced
the package manifest (but didn't, because it already existed), that
path is copied from the target which did produce the package
manifest.

(Bitbake rev: 79b8e349a0da2ea6b97ad82daa5837e6dfffe0af)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:52 +01:00
Elliot Smith 150e5588a0 bitbake: buildinfohelper: only record image files for tasks which make images
If a target is built which is classified as an "image" target
(e.g. "core-image-minimal"), Toaster reads the list of files in
the image (from the files-in-image.txt file).

However, Toaster continues to do this for builds which don't
produce images, if the recipe providing the target is an
image recipe. This can result in a list of files in the image
being attached to a target which didn't produce an image (e.g.
rootfs).

When associating files with an image, ensure that only targets
with a task which produces an image have "files in the image"
associated with them.

[YOCTO #9784]

(Bitbake rev: 44375d0c2a88e0070b8067c9285b89c54eaf3152)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00
Elliot Smith 00c2c0be5e bitbake: toaster: improve scan for SDK artifacts
SDK artifacts were previously picked up by toaster.bbclass and
notified to buildinfohelper (via toasterui). The artifacts
were then added to the Build object, so that it wasn't clear
which artifact went with which target; we were also unable
to attach SDK artifacts to a Build if they had already been
attached to a previous build.

Now, toaster.bbclass just notifies the TOOLCHAIN_OUTPUTNAME when
a populate_sdk* target completes. The scan is moved to buildinfohelper,
where we search the SDK deploy directory for files matching
TOOLCHAIN_OUTPUTNAME and attach them to targets (not builds).

If an SDK file is not produced by a target, we now look for a
similar, previously-run target which did produce artifacts.
If there is one, we clone the SDK artifacts from that target
onto the current one.

This all means that we can show SDK artifacts by target, and should
always get artifacts associated with a target, regardless of whether
it really build them.

This requires an additional model, TargetSDKFile, which tracks
the size and path of SDK artifact files with respect to Target
objects.

[YOCTO #8556]

(Bitbake rev: 5e650c611605507e1e0d1588cd5eb6535c2d34fc)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00
Elliot Smith f39ae146ea bitbake: buildinfohelper: fix retrieval of targets
When buildinfohelper records the targets for a build, it looks
up any existing targets for a build and creates them if they
are not present. This is because in the case of Toaster-triggered
builds, the Target objects have already been created (inside
triggerBuild()) and don't need to be recreated; but in the case
of cli builds, the Target objects have to be created by
buildinfohelper.

The issue is that the code for retrieving an existing target for
a build only looks for Targets with a matching target and build,
e.g. Targets for build X with target "core-image-minimal". But it
is perfectly legitimate to call bitbake with a command like
"bitbake core-image-minimal:do_populate_sdk
core-image-minimal:do_populate_sdk_ext". In such a case, the
code which looks for matching targets finds two objects, as it
doesn't filter by task.

Add the task into the filter for the Target so that only one
Target object is be returned. Note that a command
line like "bitbake recipe:task recipe:task" will still cause an
error as bitbake doesn't de-duplicate the command line arguments
and will run the recipe:task combination twice.

(Bitbake rev: 1c0a689fdaae6469d4afb98583161073d32ea50b)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00