Commit Graph

5101 Commits

Author SHA1 Message Date
Daniel Lublin 5c1ad6852e bitbake: lib/bs4: Fix imports from html5lib >= 0.9999999/1.0b8
As of html5lib 0.9999999/1.0b8 (released on July 14, 2016), some modules
have moved from _base to base. Handle this, while staying compatible
with earlier versions.

(Bitbake rev: 1679188f9c55c615cae780f2b5e6852dea9cf2ec)

Signed-off-by: Daniel Lublin <daniel@lublin.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:43:20 +00:00
Sujith H 1b4fa3f095 bitbake: cooker: convert type which needs to be marshalled
We assume that the value taken by variable v can be string,
integer or any type which can be marshalled by xmlrpc. This
change would help us to convert the non marshallable types
to string. So that we don't get exception from xmlrpc.

[YOCTO #10740]

(Bitbake rev: efb0e47479b3526bc047112f7200087c5844bba4)

Signed-off-by: Sujith H <sujith.h@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:42:22 +00:00
brian avery da22be9904 bitbake: toaster: browser tests - add Selenium Docker container as driver
Adds the ability to specify a Selenium Docker container server as
a driver. This allows for repeatable tests independent of host.
Currently we assume you are using the Firefox container. Instructions
are located in the README in tests/browser.

(Bitbake rev: 7df842f8f8b2ae640109ed06729ab59c9469fc64)

Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:42:22 +00:00
Ismo Puustinen e74831eba7 bitbake: depexp: fix string formatting.
The parameters to Python string formatting need to be inside a tuple.

(Bitbake rev: 3c82af11b89cf251c3e56725a1eed2d3f4bd835b)

Signed-off-by: Ismo Puustinen <ismo.puustinen@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:42:22 +00:00
Patrick Ohly 268bf22263 bitbake: monitordisk: add event
The current disk usage is interesting and may be worth logging over
time as part of the build statistics. Instead of re-implementing the
code and the configuration option (BB_DISKMON_DIRS), the information
gathered by monitordisk.py is made available to buildstats.bbclass via
a new event.

This has pros and cons:
- there is already a useful default configuration for "interesting" directories
- no code duplication
- on the other hand, users cannot configure recording separately from
  monitoring (probably not that important)

(Bitbake rev: f065ac17d0031dca6309ddbff18c8792630de865)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:42:22 +00:00
Patrick Ohly 091ebb8665 bitbake: monitordisk.py: minor code and comment cleanup
There's no need to encode and decode the hash key as a single string,
a tuple works just fine. Iterating over entries can be written more
concisely.

Entries in the stat results are integers, not floating point values.

(Bitbake rev: 3c943e989964382c0b819d92de26a0c914ebed33)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:42:22 +00:00
Patrick Ohly ad20ee9feb bitbake: runqueue.py: monitor disk space at regular time intervals
Hooking the disk monitor into the regular heatbeat event instead
of the runqueue solves two problems:

- When there is just one long running task which fills up the disk,
  the previous approach did not notice that until after the completion
  of the task because _execute_runqueue() only gets called on task
  state changes. As a result, aborting a build did not work in this
  case.

- When there are many short-lived tasks, disk space was getting
  checked very frequently. When the storage that is getting checked
  is on an NFS server, that can lead to noticable traffic to the
  server.

(Bitbake rev: 4547eea26803a9cd355d8b045197bcbdbb36a9ad)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:42:22 +00:00
Patrick Ohly 083365143e bitbake: cooker process: fire heartbeat event at regular time intervals
The intended usage is for recording current system statistics from
/proc in buildstats.bbclass during a build and for improving the
BB_DISKMON_DIRS implementation.

All other existing hooks are less suitable because they trigger at
unpredictable rates: too often can be handled by doing rate-limiting
in the event handler, but not often enough (for example, when there is
only one long-running task) cannot because the handler does not get
called at all.

The implementation of the new heartbeat event hooks into the cooker
process event queue. The process already wakes up every 0.1s, which is
often enough for the intentionally coarse 1s delay between
heartbeats. That value was chosen to keep the overhead low while still
being frequent enough for the intended usage.

If necessary, BB_HEARTBEAT_EVENT can be set to a float specifying
the delay in seconds between these heartbeat events.

(Bitbake rev: 7cf22ea057d28c54bd98dc1ab7a43402a29ff1f5)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:42:22 +00:00
Ross Burton 9e63f81c78 bitbake: ast: remove BBVERSIONS support
BBVERSIONS is moderately horrible and it doesn't appear to be actually used by
anyone, so remove it to simplify the finalise codepaths.

(Bitbake rev: 0bb188f01e396052b127e170a25246d79a6d6741)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:10 +00:00
Patrick Ohly caf1a69577 bitbake: codeparser.py: support deeply nested tokens
For shell constructs like
   echo hello & wait $!
the process_tokens() method ended up with a situation where "token"
in the "name, value = token" assignment was a list of tuples
and not the expected tuple, causing the assignment to fail.

There were already two for loops (one in _parse_shell(), one in
process_tokens()) which iterated over token lists. Apparently the
actual nesting can also be deeper.

Now there is just one such loop in process_token_list() which calls
itself recursively when it detects that a list entry is another list.

As a side effect (improvement?!) of the loop removal in
_parse_shell(), the local function definitions in process_tokens() get
executed less often.

Fixes: [YOCTO #10668]

(Bitbake rev: d18a74de9ac75ba32f84c40620ca9d47c1ef96a3)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:10 +00:00
Markus Lehtonen 38438b6cf4 bitbake: fetch2: obey BB_ALLOWED_NETWORKS when checking network access
[YOCTO #10508]

(Bitbake rev: ddd3bc2d64d7240ecb6b6e4a1ae29b1faef6cc22)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:10 +00:00
Michael Wood 4e48892b85 bitbake: toaster: buildinfohelper Clarify log message for build history
(Bitbake rev: 5accd6c4d1dcdf6609b4ed25c2b5e4faaf7f0909)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:10 +00:00
Michael Wood 6dba0281e1 bitbake: toaster: buildinfohelper fix _get_layer_version_for_dependency
This function is simplified by not trying to handle replacing the regex
and just compiling and using it for matching.

- Fix typo in logger output with undefined variable
- Fix pyflake errors

(Bitbake rev: ea298ece8d678889cd5bcde46e00545e9a73edb9)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:10 +00:00
Michael Wood 439f3da1a1 bitbake: toaster: buildinfohelper Simplify layer event to toaster layer function
Simplify the layer event information to layer version object in toaster
function. Previously this attempted many different methods of trying to
obtain the correct layer from toaster by manipulating the data from the
event or the data from the known layers to try and match them together.

We speed up and simplify this process by making better use of django's
orm methods and by working down the most likely matching methods in order
of accuracy.

[YOCTO #10220]

(Bitbake rev: 6935cc06974ea94c9971ede89b6e8f0eae9c195b)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:10 +00:00
Michael Wood 5de7f159a1 bitbake: toaster: bldcontrol Move CustomImageRecipe file creation into own function
Move the custom image file creation (i.e. create the layer file
structure, conf and recipe file) into it's own function and remove the
creation of the BRLayer as this is done at schedule_build just like all
the other layers.

Fix a bug where the toaster-custom-images layer was always being appened
to the layer list if the directory exists.

(Bitbake rev: 15a42b36c01fccd79e5aa0788dea5640b253982b)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:10 +00:00
Michael Wood 6f8df05e6e bitbake: toaster: orm models Handle CustomImageRecipe BRLayer here
The schedule_build function on the project object is where the BRLayers
are created for the build. Instead of creating the BRLayer for the
CustomImageRecipe in the localhostbbcontroller create it here so that
all that mechanism is in one place.

Also fix a number of pyflake errors.

(Bitbake rev: f8d3ea784937b6e416d3e5a4feb1283c478e4caa)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:10 +00:00
Michael Wood b58b982eaf bitbake: toaster: orm models Project class Fix pyflake errors
(Bitbake rev: 69f33397083f54f977fa0cd4dd621731f32fd034)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:10 +00:00
Michael Wood 30a9f65dcc bitbake: toaster: buildinfohelper toaster-custom-images layer
This fixes the unidentified layers issue by making the
toaster-custom-images layer a local layer. By doing this we also fix the
git assumptions made for the local layers which stop recipes and other
meta data being associated with them. This also removed some of the
special casing previously needed when we didn't have the concept of a
local (non git) layer.

Also rename created flag var to a have a different var for each returned
value so that the same value isn't used multiple times.

[YOCTO #10220]

(Bitbake rev: ba5332d4960d7f4f79aef63136796e2fa67284e3)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:10 +00:00
Sujith H fa5ea98d3b bitbake: toaster: localhostbecontroller write toaster layers for project to toaster-bblayers.conf
Instead of updating conf/bblayers, here we update toaster-bblayers.conf
file. So extra effort to update bblayers.conf can be removed safely.

(Bitbake rev: f3e99d820f3798869a2a1d1604709c1c324dbbab)

Signed-off-by: Sujith H <sujith.h@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:10 +00:00
Sujith H 5f0405abed bitbake: toaster: localhostbecontroller accept custom init script for build
When passed variable CUSTOM_BUILD_INIT_SCRIPT to toaster
setting, it would be nice to use it. Else toaster
can use oe-init script. This gives an oppurtunity to
use customized build init scritps.

(Bitbake rev: 9d168bb5f05453bdb7156793eea25da0a3119b4d)

Signed-off-by: Sujith H <sujith.h@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:10 +00:00
Reyna, David c57533b83b bitbake: toaster: tablejs Fix missing close square bracket
There is a missing close square bracket.

[YOCTO #10631]

(Bitbake rev: f3da9f85e6036e6f43377172cbcfe701e0efca7f)

Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:10 +00:00
Reyna, David 13c0ffc81c bitbake: toaster: orm gen_layerdeps Protect against circular Layer dependencies
Limit the recursion (to say 20 levels) when processing layer dependencies
so that circular dependecies do not cause infinite decent and an
out-of-memory failure. The duplicate found layers are already immediately
filtered in the code.

[YOCTO #10630]

(Bitbake rev: e9efef0bdb8068984c3013b87aac9e872ffb38ae)

Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:10 +00:00
Michael Wood e336017e87 bitbake: toaster: customrecipejs Consume click event on 'a' link if disabled
Consume the click event on the download recipe link if it's disabled. To
prevent the link from sending user to an error page.
See http://getbootstrap.com/css/#forms-disabled-fieldsets and a link
caveat.

[YOCTO #10151]

(Bitbake rev: bc8401e78cea140349bded228d38f72f628b3980)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:09 +00:00
Michael Wood 6191f71f81 bitbake: toaster: Add an example production settings file
Add an example settings that can be used for the basis of the production
instance of Toaster.

[YOCTO #10581]

(Bitbake rev: 1a7c356491b88c8decced39fb2039ef90065f2d2)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:09 +00:00
Michael Wood 4477a762bd bitbake: toaster: tests Add management command tests
Add some simple sanity tests for the management commands that we use for
Toaster.

Can be executed with ./manage.py test tests.commands

For faster execution use the test settings and keepdb flag:
DJANGO_SETTINGS_MODULE=toastermain.settings_test ./manage.py test
tests.commands --keepdb

(Bitbake rev: 161ea71519e7f70d4aadaafc9c3294a12612f0cb)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:09 +00:00
Michael Wood 310a9e5d35 bitbake: toaster: runbuilds Write the pidfile in python rather than shell script
Write the pid file out in the start up of this management command. This
ensures this has happened instead of relying on the shell command having
been run which may or may not be the case. This also makes it simpler for
testing.

Couple of clean ups of runbuilds as identified by pyflake

(Bitbake rev: 999e980ee1a58d16f33ef6c0e41aecdcd0206f39)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:09 +00:00
Joshua Lock ddaac5e4e3 bitbake: bitbake: remove True option to getVarFlag calls
getVarFlag() now defaults to expanding by default, thus remove the
True option from getVarFlag() calls with a regex search and
replace.

Search made with the following regex:
getVarFlag ?\(( ?[^,()]*, ?[^,()]*), True\)

(Bitbake rev: c19baa8c19ea8ab9b9b64fd30298d8764c6fd2cd)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:09 +00:00
Joshua Lock 1fce7ecbbb bitbake: bitbake: remove True option to getVar calls
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.

Search made with the following regex: getVar ?\(( ?[^,()]*), True\)

(Bitbake rev: 3b45c479de8640f92dd1d9f147b02e1eecfaadc8)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:09 +00:00
Richard Purdie 3d34ae6c11 bitbake: bitbake-worker: Handle cooker/worker IO deadlocking
I noiced builds where tasks seemed to be taking a surprisingly long time.
When I looked at the output of top/pstree, these tasks were no longer
running despite being listed in knotty. Some were in D/Z state waiting for
their exit code to be collected, others were simply not present at all.

strace showed communication problems between the worker and cooker, each
was trying to write to the other and nearly deadlocking. Eventually, timeouts
would allow them to echange 64kb of data but this was only happening every
few seconds.

Whilst this particularly affected builds on machines with large numbers
of cores (and hence highly parallal task execution) and in cases where
I had a lot of debug enabled, this situation is clearly bad in general.

This patch introduces a thread to the worker which is used to write data
back to cooker. This means that the deadlock can't occur and data flows
much more freely and effectively.

(Bitbake rev: 3cb0d1c78b4c2e4f251a59b86c8da583828ad08b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-29 11:22:06 +00:00
Kai Kang cd381cd714 bitbake: COW.py: fix sample codes
The call of methods iteritems() and itervalues() in sample codes were
replaced by items() and values() to convert to Python 3 by Bitbake rev
d0f904d407f57998419bd9c305ce53e5eaa36b24. But the methods iteritems()
and itervalues() belong to class COWDictMeta not class dict or set. The
modifications should not be made in purpose that it fails to run sample
codes, so revert them.

(Bitbake rev: d140f0ee6f301264e226914766d9f63558acfd6c)

Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 10:59:56 +00:00
Michael Wood dbf3a9fa8e bitbake: toaster: layerindex updater Take into account layers being predefined
As we can now provide layer definitions through fixtures we need to be
more clever how we update the metadata in the database to avoid
duplicate metadata being created. To do this we make more effort to
match existing data in the database and update only the fields which
will be better provided by the layer index.

This removes the need for us to special case layers which are provided
as part of poky such as openembedded-core or meta-poky which exist on
the layerindex but with different git urls.

(Bitbake rev: f981b68f66718d5b196684f4e378a5f195ff0337)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 10:59:56 +00:00
Michael Wood f59ac87b46 bitbake: toaster: orm/fixtures Add the master release and correct morty release
Add the master release option to base your project on and correct the
morty release so that for poky based setups we use the poky provided
version of the layer rather than checking out the layer from its own git
repository.

[YOCTO #10497]

(Bitbake rev: c83ab92362378b22d3f4d6119bf362f704577ca2)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 10:59:56 +00:00
Michael Wood 50cb0f596f bitbake: toaster: settings fixture Set default release to master
Now that morty has been released we now set the DEFAULT_RELEASE back to
master.

(Bitbake rev: 115cee16a9aecfcd1061bb106ebca4e861d9b296)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 10:59:56 +00:00
Patrick Ohly 7bb38a3358 bitbake: taskdata.py: improve handling of depends/rdepends
Error handling only caught the cause where a dependency did not have
any colon, but ignored the case where more than one was given. Now
"pn:task:garbage" will raise an error instead of ignoring ":garbage".

The error message had a misplaced line break (?) with the full stop
on the next line. Indenting the explanation with a space might have
been intended and is kept.

split() was called three times instead of just once.

Instead of improving the two instances of the code (one for 'depends',
one for 'rdepends'), the common code is now in a helper function.

(Bitbake rev: 063d255fdcb3f79b2d1b0badedc80384b295a3f5)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 10:59:56 +00:00
Patrick Ohly d7f76363fb bitbake: data_smart.py: don't reorder internal bitbake variables when calculating hash
Commit 260ced745 added __BBTASKS, __BBANONFUNCS, __BBHANDLERS to the
data that gets hashed, but only after reordering these lists. The
intention probably was to make the hash deterministic, but that's
unnecessary (the content of the variables should already be
deterministic) and hides potential reasons that might require
re-parsing.

(Bitbake rev: 3511d464f3a9d8b4334cda384b35016de69ce49e)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 10:59:56 +00:00
Richard Purdie 2631c375b8 bitbake: data_smart: Default to expansion for getVar/getVarFlags
We've been building to this for a while, default to return expanded
values for getVar/getVarFlags.

We can then go through and remove the "True" option to many of the
calls to this function, all function calls should have a default by now
though since the parameter has been required for a while.

(Bitbake rev: caf5bb9b7fe254bca9da077ebcb84a37d1f96dd4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 10:59:56 +00:00
Richard Purdie 30205c7148 bitbake: data: Drop deprecated old style bitbake API
The old style bb.data.getVar/setVar API has long since been deprecated in
favour of d.getVar/setVar and friends.

Now we're about to change the default expansion parameter, drop the old APIs
to simplify the transition and ensure everyone is using the new style functions.
Conversion is trivial if there are remaining stragglers.

I've left bb.data.expand() for now since its more widely used but would make a good
follow up patch series.

(Bitbake rev: 1825604d46fcd29fad6cfd325f1cb1e1b457d2c9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 10:59:56 +00:00
Richard Purdie 8de811ae76 bitbake: lib/bb: Don't use deprecated bb.data.getVar/setVar API
The old style bb.data.getVar/setVar API is obsolete. Most of bitbake
doesn't use it but there were some pieces that escaped conversion. This
patch fixes the remaining users mostly in the fetchers.

(Bitbake rev: ff7892fa808116acc1ac50effa023a4cb031a5fc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 10:59:56 +00:00
Paul Eggleton 4d8ee55164 bitbake: fetch2: npm: conditionally hide NPM_LOCKDOWN / NPM_SHRINKWRAP warnings
If ud.ignore_checksums is set (which we currently use to suppress the
warnings for missing SRC_URI checksums when fetching files from
scripts), then if we're fetching an npm package we should similarly
suppress the warnings when NPM_LOCKDOWN and NPM_SHRINKWRAP aren't set.

At the same time, make any errors reading either of these files actual
errors since if the file is specified and could not be found, that
should be an error - not the exact same warning.

Fixes [YOCTO #10464].

(Bitbake rev: cefb8c93c8299e68352cf7ec5ad9ca50c0d499ed)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 10:59:56 +00:00
brian avery 7c3a47ed89 bitbake: toaster: settings set ALLOWED_HOSTS to * in debug mode
As of Django 1.8.16, Django is rejecting any HTTP_HOST header that is
not on the ALLOWED_HOST list.  We often need to reference the
toaster server via a fqdn, if we start it via webport=0.0.0.0:8000 for
instance, and are hitting the server from a laptop. This change does
reduce  the protection from a DNS rebinding attack, however, if you are
running the toaster server outside a protected network, you should be
using the production instance.

[YOCTO #10578]

(Bitbake rev: 7f51149453c96a3f1da64ea85306518fd2b65f21)

Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-16 11:38:44 +00:00
Robert Yang edb5b524b3 bitbake: lib/bb/cooker.py: fix for BBFILE_PATTERN = ""
There would be error when BBFILE_PATTERN = None:
  BBFILE_PATTERN_foo not defined

This is the correct behaviour, but when the layer sets BBFILE_PATTERN = "",
it would match all the remaining recipes, and cause "No bb files matched BBFILE_PATTERN"
warnings for all the layers which behind it.

When a layer sets BBFILE_PATTERN = "" (for example, a layer only
provides git repos and source tarballs), now it means has no recipes.
This is different from BBFILE_PATTERN_IGNORE_EMPTY, the later one means
that it *may* not have any recipes.

(Bitbake rev: 91c3b34625fac2a0f093a4b46a46e89f813e7972)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-04 12:50:56 +00:00
Olaf Mandel 0038120f05 bitbake: toasterui.py: warn if buildstats is missing
Like for buildhistory, warn if buildstats is missing from INHERIT.

CC: Joshua Lock <joshua.g.lock@linux.intel.com>
(Bitbake rev: 3570a8cf94354c8ab07513c304ebae33623fea33)

Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-04 12:50:56 +00:00
Ed Bartosh b8cabc7ed3 bitbake: toaster: add tests/eventreplay/README
Put instructions on how to prepare event log files
and run eventreplay tests.

(Bitbake rev: 0e675547166acc8650498e153bd3482420342c32)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-04 12:50:56 +00:00
Ed Bartosh 934c9efa66 bitbake: toaster: add eventreplay test case for zlib
Run toaster-eventreplay with zlib.events.
Check if zlib build and package present in Toaster database.

(Bitbake rev: c922f4904301174cc72ba35e76870fbf964082cf)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-04 12:50:56 +00:00
Ed Bartosh 3503c5426e bitbake: toaster: add eventreplay test case for core-image-minimal
Run toaster-eventreplay with core-image-minimal.events and
test if all required packages present in Target_Installed_Package
table.

(Bitbake rev: 73410e6dc965b2885c68e87ed6fa8d8b57e9c49d)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-04 12:50:56 +00:00
Ed Bartosh 98ac0b0a5b bitbake: toaster: use current directory if BUILDDIR is not set
If BUILDDIR environment variable is not set signal_runbuilds function
throws TypeError as os.getenv('BUILDDIR') returns None:

ERROR: unsupported operand type(s) for +=: 'NoneType' and 'str'
Traceback (most recent call last):
  File "bitbake/lib/bb/ui/toasterui.py", line
391, in main
    buildinfohelper.update_build_information(event, errors, warnings,
taskfailures)
  File "bitbake/lib/bb/ui/buildinfohelper.py",
line 1184, in update_build_information
    self.internal_state['build'], errors, warnings, taskfailures)
  File "bitbake/lib/bb/ui/buildinfohelper.py",
line 238, in update_build_stats_and_outcome
    signal_runbuilds()
  File "bitbake/lib/toaster/orm/models.py", line
1746, in signal_runbuilds
    '.runbuilds.pid')) as pidf:
  File "/usr/lib64/python3.4/posixpath.py", line 82, in join
    path += b
TypeError: unsupported operand type(s) for +=: 'NoneType' and 'str'

Used os.getenv('BUILDIR', '.') to make it always return meaningful
directory path. Current directory '.' will be used if BUILDDIR is
not set.

(Bitbake rev: da631152a0db3f432709a05ff15a268d784ca3ab)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-04 12:50:55 +00:00
brian avery 637c93edfa bitbake: toaster: buildinfohelper Handle regex paths
We were presuming that all the layer dependency information was of the
form "^/path/to/layer" to we were just stripping the leading "^" off of
the layer information when we were matching the layer priorities to the
toaster database.  This patch splits out the priorities layer match which
gets a  regex from the task/recipe match which is gets a path.

(Bitbake rev: e23b574fe52f416184ee43838b8ab28b5b8eb71d)

Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-04 12:50:55 +00:00
Michael Wood 754fa212bc bitbake: toaster: tests builds test_core_image_min Clean ups
Instead of searching for the build for each test just use the returned
value of the completed build.

(Bitbake rev: ecb94e50262b3f4ca8d5107f77f053335ef23511)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-04 12:50:55 +00:00
Michael Wood 83ebb89877 bitbake: toaster: tests builds Update buildtest
Now that we're using fixtures for configuration just load these instead
of trying to search for a toasterconf json file.

Also for convenience add the ability for the tests to source the build
environment script. To use this test make sure that directories are in
the same layout as poky.

(Bitbake rev: 448d1d9dc8989ef4c997a90c71cd7e1da0495c1c)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-04 12:50:55 +00:00
Michael Wood 8e70fa1d79 bitbake: toaster: orm models Handle run builds process not yet running
During tests we may want to call the runbuilds process manually for
example when doing a "one shot" approach rather than a long running
process during tests.

(Bitbake rev: 60d3f93836da5523705b0b2e25567d1c9040ec89)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-04 12:50:55 +00:00