As of html5lib 0.9999999/1.0b8 (released on July 14, 2016), some modules
have moved from _base to base. Handle this, while staying compatible
with earlier versions.
(Bitbake rev: 1679188f9c55c615cae780f2b5e6852dea9cf2ec)
Signed-off-by: Daniel Lublin <daniel@lublin.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We assume that the value taken by variable v can be string,
integer or any type which can be marshalled by xmlrpc. This
change would help us to convert the non marshallable types
to string. So that we don't get exception from xmlrpc.
[YOCTO #10740]
(Bitbake rev: efb0e47479b3526bc047112f7200087c5844bba4)
Signed-off-by: Sujith H <sujith.h@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds the ability to specify a Selenium Docker container server as
a driver. This allows for repeatable tests independent of host.
Currently we assume you are using the Firefox container. Instructions
are located in the README in tests/browser.
(Bitbake rev: 7df842f8f8b2ae640109ed06729ab59c9469fc64)
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The parameters to Python string formatting need to be inside a tuple.
(Bitbake rev: 3c82af11b89cf251c3e56725a1eed2d3f4bd835b)
Signed-off-by: Ismo Puustinen <ismo.puustinen@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current disk usage is interesting and may be worth logging over
time as part of the build statistics. Instead of re-implementing the
code and the configuration option (BB_DISKMON_DIRS), the information
gathered by monitordisk.py is made available to buildstats.bbclass via
a new event.
This has pros and cons:
- there is already a useful default configuration for "interesting" directories
- no code duplication
- on the other hand, users cannot configure recording separately from
monitoring (probably not that important)
(Bitbake rev: f065ac17d0031dca6309ddbff18c8792630de865)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There's no need to encode and decode the hash key as a single string,
a tuple works just fine. Iterating over entries can be written more
concisely.
Entries in the stat results are integers, not floating point values.
(Bitbake rev: 3c943e989964382c0b819d92de26a0c914ebed33)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Hooking the disk monitor into the regular heatbeat event instead
of the runqueue solves two problems:
- When there is just one long running task which fills up the disk,
the previous approach did not notice that until after the completion
of the task because _execute_runqueue() only gets called on task
state changes. As a result, aborting a build did not work in this
case.
- When there are many short-lived tasks, disk space was getting
checked very frequently. When the storage that is getting checked
is on an NFS server, that can lead to noticable traffic to the
server.
(Bitbake rev: 4547eea26803a9cd355d8b045197bcbdbb36a9ad)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The intended usage is for recording current system statistics from
/proc in buildstats.bbclass during a build and for improving the
BB_DISKMON_DIRS implementation.
All other existing hooks are less suitable because they trigger at
unpredictable rates: too often can be handled by doing rate-limiting
in the event handler, but not often enough (for example, when there is
only one long-running task) cannot because the handler does not get
called at all.
The implementation of the new heartbeat event hooks into the cooker
process event queue. The process already wakes up every 0.1s, which is
often enough for the intentionally coarse 1s delay between
heartbeats. That value was chosen to keep the overhead low while still
being frequent enough for the intended usage.
If necessary, BB_HEARTBEAT_EVENT can be set to a float specifying
the delay in seconds between these heartbeat events.
(Bitbake rev: 7cf22ea057d28c54bd98dc1ab7a43402a29ff1f5)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Setting WKS_FILE variable in qemux86-64 made wic test to
use wrong wks file to produce an image and resulted in
test_qemu failure.
Used conditional assignment in qemux86-64 and explicitly
set WKS_FILE in wic testing suite to make the suite to use
wic-image-minimal.wsk. This should fix test_qemu failure.
(From OE-Core rev: 3bca4d18c2712e3b154bacfb917f0a749ebaddeb)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The do_deploy runs depmodwrapper which requires
pkgdata/kernel-depmod/kernel-abiversion, which is generated by
PACKAGEFUNCS. So we need let do_deploy depend on do_packagedata.
This can fix the errors which happens sometimes when kernel upgrades:
DEBUG: Executing shell function do_deploy
Error: Kernel version 4.8.8-WR9.0.0.1_standard does not match kernel-abiversion (4.8.8-WR9.0.0.0_standard)
And we only see this error when kernel upgrades and rebuild, but doesn't
see it in a normal build, this is becuase depmodwrapper doesn't exit
error when kernel-depmod/kernel-abiversion doesn't exit, it just prints
an error which should go into log.do_deploy:
if [ ! -r /path/to/sysroots/qemux86-64/pkgdata/kernel-depmod/kernel-abiversion ]; then
echo "Unable to read: /path/to/sysroots/qemux86-64/pkgdata/kernel-depmod/kernel-abiversion" >&2
else
[foo]
fi
We can see that there is no "exit 1", I guess it was designed to let it
can run without kernel-abiversion
(From OE-Core rev: 64148a30397b8c92414262c0d414d103d2b97ddd)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
While I couldn't find explicit documentation, it appears
that the list of MACHINEOVERRIDES should be sorted from
less specific to more specific left to right, so that
more specific overrides take precedence.
(From OE-Core rev: 59c724db5628775e77fa090183897c6ae0fdf9a8)
Signed-off-by: André Draszik <adraszik@tycoint.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Required for meta-swupd performance enhancements: in meta-swupd, the
so called "mega" image contains a rootfs with all files that can
potentially be installed on a device. Other virtual image recipes need
a subset of those files or directories, and a partial extraction from
a single tar archive is faster than letting all virtual image recipes
share access to a directory under a single pseudo instance.
It may be necessary to extract a directory with all of its attributes
without the content of the directory, hence this patch. Upstream
agreed to consider merging such a patch (see
https://groups.google.com/forum/#!topic/libarchive-discuss/JO3hqSaAVfs)
but has been slow in actually commenting on it, so for now it has
to be carried as distro patch.
(From OE-Core rev: 53126f0fbc63148cc5b22605ffbdf54d0610b545)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is a simultaneous upgrade for python 2 and 3 over setuptools.
(From OE-Core rev: f0c75fecdf15453c5fb252eb9f3de3ea9ce5169b)
Signed-off-by: Edwin Plauchu <edwin.plauchu.camacho@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently run-postinsts script has code to run postinst scripts
via opkg/dpkg configure but that code is never used. The advantage
of using package managers instead of just executing the scripts is
to keep the package manager DB updated.
Fix the script so that the package managers are used when appropriate.
Also use $localstatedir for the opkg runtime file location.
Fixes [YOCTO #10478].
(From OE-Core rev: b645919f173512f9e75aeb26348d60b63dcdc53c)
Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
TUNE_FEATURES is include in BUILDCFG_VARS, so any whitespace is
visible to the user during the build process. Remove the extra
whitespace added during the 2.1 development cycle:
http://git.openembedded.org/openembedded-core/commit/?id=f774b44fa007a2a756ada892ede832b1251d940c
For consistency, squash whitespace within PACKAGE_EXTRA_ARCHS strings
too.
(From OE-Core rev: 5610c6397ee098dd998b7417b343494de77179f9)
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Pre-processing /proc data during the build considerably reduces the
amount of data written to disk: 176KB instead of 4.7MB for a 20
minuted build. Parsing also becomes faster.
buildstats.bbclass only writes the reduced logs now, but support for
the full /proc files is kept around as reference.
(From OE-Core rev: b5e47df9af1ebbb477074587fdeae17eb2f55582)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Pre-processing /proc data during the build considerably reduces the
amount of data written to disk: 176KB instead of 4.7MB for a 20
minuted build. Parsing also becomes faster.
The disk monitor log added another 16KB in that example build. The
overall buildstat was 20MB, so the overhead for monitoring system
utilization is small enough that it can be enabled by default.
(From OE-Core rev: b17812385cd55e81066d3ceda92dffdc6e5564da)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The internal representation after parsing now matches exactly
what the drawing code needs, thus speeding up drawing a bit.
However, the main motivation is to store exactly that required
information in a more compact file.
(From OE-Core rev: ca06e67a0bb5820b38fda4c8dfee20764c1e59ae)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds a new, separate chart showing the amount of disk space used
over time for each volume monitored during the build. The hight of the
graph entries represents the delta between current usage and minimal
usage during the build.
That's more useful than showing just the current usage, because then a
graph showing changes in the order of MBs in a volume that is several
GB large would be just flat.
The legend shows the maximum of those deltas, i.e. maximum amount of
space needed for the build. Minor caveat: sampling of disk space usage
starts a bit later than the initial task, so the displayed value may
be slightly lower than the actual amount of space needed because
sampling does not record the actual initial state.
(From OE-Core rev: 263d189d066b578debf08b2bd07494a69b70f70d)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When matching fails, m.group(0) is invalid and can't be used in the
error message.
(From OE-Core rev: ddfea21e06a2e6b1a1e766969f8c134a6de7388a)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Hooks into the new monitordisk.py event and records the used space for
each volume. That is probably the only relevant value when it comes to
visualizing the build and recording more would only increase disk
usage.
(From OE-Core rev: 21a5b569370f47cc02291e1d8b76fe43faa04ea6)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The only real change is the addition of two if checks that skips the
corresponding drawing code when there is no data.
(From OE-Core rev: 1658fd5e9ca1ba793cae604c2a395d54e3ec9056)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This enables rendering of the original bootchart charts for CPU, disk
and memory usage. It depends on the /proc samples recorded by the
updated buildstats.bbclass. Currently, empty charts CPU and disk usage
charts are drawn if that data is not present; the memory chart already
gets skipped when there's no data, which will also have to be added
for the other two.
(From OE-Core rev: 233d3e50b361feea07803a9c0f2a691e687c6cd5)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code did not handle x scaling correctly when drawing starts at
some time larger than zero, i.e. it worked for normal bootchart data,
but not for the system statistics recorded by buildstats.bbclass.
(From OE-Core rev: 166f8f9aaa1f01fc6d6a5451f8f06b815c51ffae)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Substracting curr_y when determining the hight of the process chart is
wrong because the height is independent of the position where the
chart is about to be drawn. It happens to work at the moment because
curr_y is always 10 when render_processes_chart() gets called. But it
leads to a negative height when other charts are drawn above it, and
then the grid gets drawn on top of those other charts.
Substracting some constant is relevant because otherwise the box is
slightly larger than the process bars. Not sure exactly where that
comes from (text height?); leg_s seems a suitable constant and happens
to be 10, so everything still gets rendered exactly as before.
(From OE-Core rev: b6bb690728c329ae448f89a1b68298c6dd8a573a)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
/proc/[diskstats|meminfo|stat] get sampled and written to the same
proc_<filename>.log files as during normal bootchat logging. This will
allow rendering the CPU, disk and memory usage charts.
Right now sampling happens once a second, triggered by the heartbeat
event.That produces quite a bit of data for long builds, which will be
addressed in a separate commit by storing the data in a more compact
form.
(From OE-Core rev: 6f4e8180b5b4857eaf6caf410fd3a4a41ed85930)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The warning occurs when the GPT image is not the same size than the
media into which it's being flashed, causing the backup GPT table
not being at the end of the disk. However, this is expected as the
image is created before having the information about the destination
media. The error is harmless, so it will be whitelisted.
Fixes [YOCTO 10481].
(From OE-Core rev: 5cc5cdc788308a79f8f0706e6d794c602ef427ed)
Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The iwlwifi module of any given kernel has a minimum and maximum
supported firmware version. The kernel begins by attempting to load the
maximum version, and decrements until it is successful. The 4.8 kernel's
maximum supported firmware version is 24, but thus far only 22 has been
released, meaning we get errors for 24 and 23.
Filter out iwlwifi firmware load error messages, as they are not
necessarily indicative of real problems.
(From OE-Core rev: 7df570c2310efac8f9898da15deaac2b7df16655)
Signed-off-by: California Sullivan <california.l.sullivan@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When creating a patch set with cover letter using the
send-pull-request script, both the "In-Reply-To" and "References"
headers are appended twice in patch 2 and subsequent.
That's because git-format-patch already inserted them and then
git-send-email repeats that. Suppressing mail threading in
git-send-email with --no-thread avoids the problem and is the
right solution because it works regardless whether git-send-email is
called once or twicee.
Repeating these headers is a violation of RFC 2822 and can confuse
mail programs. For example, Patchwork does not detect a patch series
problem when there are these extra headers.
[YOCTO #10718]
(From OE-Core rev: 303a1aa3df43eb0b693d8602062fa33c4a08fdd6)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BBVERSIONS is moderately horrible and it doesn't appear to be actually used by
anyone, so remove it to simplify the finalise codepaths.
(Bitbake rev: 0bb188f01e396052b127e170a25246d79a6d6741)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For shell constructs like
echo hello & wait $!
the process_tokens() method ended up with a situation where "token"
in the "name, value = token" assignment was a list of tuples
and not the expected tuple, causing the assignment to fail.
There were already two for loops (one in _parse_shell(), one in
process_tokens()) which iterated over token lists. Apparently the
actual nesting can also be deeper.
Now there is just one such loop in process_token_list() which calls
itself recursively when it detects that a list entry is another list.
As a side effect (improvement?!) of the loop removal in
_parse_shell(), the local function definitions in process_tokens() get
executed less often.
Fixes: [YOCTO #10668]
(Bitbake rev: d18a74de9ac75ba32f84c40620ca9d47c1ef96a3)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This function is simplified by not trying to handle replacing the regex
and just compiling and using it for matching.
- Fix typo in logger output with undefined variable
- Fix pyflake errors
(Bitbake rev: ea298ece8d678889cd5bcde46e00545e9a73edb9)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Simplify the layer event information to layer version object in toaster
function. Previously this attempted many different methods of trying to
obtain the correct layer from toaster by manipulating the data from the
event or the data from the known layers to try and match them together.
We speed up and simplify this process by making better use of django's
orm methods and by working down the most likely matching methods in order
of accuracy.
[YOCTO #10220]
(Bitbake rev: 6935cc06974ea94c9971ede89b6e8f0eae9c195b)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move the custom image file creation (i.e. create the layer file
structure, conf and recipe file) into it's own function and remove the
creation of the BRLayer as this is done at schedule_build just like all
the other layers.
Fix a bug where the toaster-custom-images layer was always being appened
to the layer list if the directory exists.
(Bitbake rev: 15a42b36c01fccd79e5aa0788dea5640b253982b)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The schedule_build function on the project object is where the BRLayers
are created for the build. Instead of creating the BRLayer for the
CustomImageRecipe in the localhostbbcontroller create it here so that
all that mechanism is in one place.
Also fix a number of pyflake errors.
(Bitbake rev: f8d3ea784937b6e416d3e5a4feb1283c478e4caa)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fixes the unidentified layers issue by making the
toaster-custom-images layer a local layer. By doing this we also fix the
git assumptions made for the local layers which stop recipes and other
meta data being associated with them. This also removed some of the
special casing previously needed when we didn't have the concept of a
local (non git) layer.
Also rename created flag var to a have a different var for each returned
value so that the same value isn't used multiple times.
[YOCTO #10220]
(Bitbake rev: ba5332d4960d7f4f79aef63136796e2fa67284e3)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of updating conf/bblayers, here we update toaster-bblayers.conf
file. So extra effort to update bblayers.conf can be removed safely.
(Bitbake rev: f3e99d820f3798869a2a1d1604709c1c324dbbab)
Signed-off-by: Sujith H <sujith.h@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When passed variable CUSTOM_BUILD_INIT_SCRIPT to toaster
setting, it would be nice to use it. Else toaster
can use oe-init script. This gives an oppurtunity to
use customized build init scritps.
(Bitbake rev: 9d168bb5f05453bdb7156793eea25da0a3119b4d)
Signed-off-by: Sujith H <sujith.h@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a missing close square bracket.
[YOCTO #10631]
(Bitbake rev: f3da9f85e6036e6f43377172cbcfe701e0efca7f)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Limit the recursion (to say 20 levels) when processing layer dependencies
so that circular dependecies do not cause infinite decent and an
out-of-memory failure. The duplicate found layers are already immediately
filtered in the code.
[YOCTO #10630]
(Bitbake rev: e9efef0bdb8068984c3013b87aac9e872ffb38ae)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Consume the click event on the download recipe link if it's disabled. To
prevent the link from sending user to an error page.
See http://getbootstrap.com/css/#forms-disabled-fieldsets and a link
caveat.
[YOCTO #10151]
(Bitbake rev: bc8401e78cea140349bded228d38f72f628b3980)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Daniel Lublin
Sujith H
brian avery
Ismo Puustinen
Patrick Ohly
Ed Bartosh
Huang Qiyu
Chen Qi
Linus Svensson
Robert Yang
André Draszik
Khem Raj
Edwin Plauchu
Jussi Kukkonen
Andre McCurdy
Jair Gonzalez
California Sullivan
Ross Burton
Markus Lehtonen
Michael Wood
Reyna, David