There is some probability (depends on system load) to get empty
or line containing "0" as the last line while reading /proc/PID/io.
Avoid build failure by checking if line contains separator
":" before split.
(From OE-Core rev: b26feaf51af55f17fad79dbd53dd3ec0a37c38ff)
Signed-off-by: Pavel Modilaynen <pavelmn@axis.com>
Signed-off-by: Daniel Lublin <daniel@lublin.se>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're placing an object into the datastore - it's very definitely not
something we want to be expanding.
(From OE-Core rev: adfee41d2b1edbd2abd09f9101c359919e21e5d6)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.
Search made with the following regex: getVar ?\(( ?[^,()]*), True\)
(From OE-Core rev: 7c552996597faaee2fbee185b250c0ee30ea3b5f)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Hooks into the new monitordisk.py event and records the used space for
each volume. That is probably the only relevant value when it comes to
visualizing the build and recording more would only increase disk
usage.
(From OE-Core rev: 21a5b569370f47cc02291e1d8b76fe43faa04ea6)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
/proc/[diskstats|meminfo|stat] get sampled and written to the same
proc_<filename>.log files as during normal bootchat logging. This will
allow rendering the CPU, disk and memory usage charts.
Right now sampling happens once a second, triggered by the heartbeat
event.That produces quite a bit of data for long builds, which will be
addressed in a separate commit by storing the data in a more compact
form.
(From OE-Core rev: 6f4e8180b5b4857eaf6caf410fd3a4a41ed85930)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
All lines except one (the one containing the 'Elapsed Time') follows the format
'stat: value'. Fix that so post parsing the stats is simpler.
(From OE-Core rev: 47dfd172754440a7ab28eb1b59d9bd484cdad22a)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After 0d6b7276003f1afabc6de683f663540327d52bdc, the exceptions are
correctly checked if the rootfs size check fails. In case of a
failure a build error is triggered.
However, there are cases where this is known to fail (e.g.,
with meta-swupd the rootfs for swupd images is other than IMAGE_ROOTFS).
Because of that, check IMAGE_ROOTFS exists before trying to get the
size of it. Also, in case of any error catched as err, simply print
out a warning.
(From OE-Core rev: b4929542ff01a24bea5edd1c40e3174f55e213ff)
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This begins moving away from the deprecated subprocess calls in an
effort to eventually move to some more global abstraction using the run
convenience method provided in python 3.5.
[ YOCTO #9342 ]
(From OE-Core rev: 0d6b7276003f1afabc6de683f663540327d52bdc)
Signed-off-by: Stephano Cetola <stephano.cetola@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a failure occurs early in the task, its possible we can have a
TaskFailed before the TaskStarted event can be triggered. This in
turn causes another traceback as the directory buildstats writes
files into doesn't exist.
Ensure the directory exists so we can see the original error.
(From OE-Core rev: bd0a65ec47ebf55c549c9ef276b201b72396ce2b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is not guaranteed Linux kernel was configured with process I/O
statistics enabled. If process I/O statistcs are not present, issue
a one time warning and do not attempt to read the non-existing stats
counters.
[YOCTO#9025]
(From OE-Core rev: b39e84edb02d03102b9a571c21e5328c159c4378)
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add IO stats and getrusage() data to the task statistics. We
also drop the CPU percentage calculation since its pretty arbitrary
and not very accurate/useful.
In particular we can now see the user and sys times as well as the
wall clock times.
(From OE-Core rev: b849130f71d3ba32a6fa94c291ca6ce7c7c3b3d1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The combined build and task data code makes changing things hard, separate
out the functions so that changes can be made to the task data whilst the
build data remains unchanged.
(From OE-Core rev: c79cfce4f820f20346d0565df8df626832976e28)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than bb.data and e.data, cleanup to use 'd' and match the
standard coding style.
(From OE-Core rev: dbce2c6122d492ba86873db4d02322f0df7a6752)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current setting and getting of the "name" to use for buildstats is
convoluted and not particularly interesting. We only need this for the
e.getPkgs()[0] component of the path which is the first target listed
on the commandline. This is pretty arbitrary.
If we drop that piece, we can assume BUILDNAME is common for all events
and simply use this and query it. If BUILDNAME did change, that would
be a bug and it should be fixed elsewhere.
Also take the opportunity to share some common code since the function
now has the eventmask.
(From OE-Core rev: 918d83460639df273f38ae079ffeebd6a79b3373)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The existing diskstats data from buildstats simply isn't useful. It
gives stats on the total IO counts on some random disk within the
system. This means that the count includes data from all other tasks
running at the same time and from any other process running on the
system.
I've been unable to find any use for the data so as a start at
reworking the class, remove the related code.
(From OE-Core rev: f9fb02909f332365cad329352956a29cff6eba77)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Rename BNFILE and DEVFILE so that they are clearly specific to this
class, since they end up in the global scope
* Use "with open" when opening files consistently
* Use getVar('PF', True) instead of expand('${PF}')
* Drop some unnecessary assignments
(From OE-Core rev: 2392c9bae9fc7732ebafb8c2cd42e49ab281afc9)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
mkdirhier() doesn't raise an exception if the directory exists, so if it
does raise one we should just be failing, otherwise we're just going
to hit errors later anyway.
(From OE-Core rev: 76daab82185851608e5e225487e411504ecb6569)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Drop trailing whitespace
* Use spaces around equals in assignments
* Replace an errant tab with spaces
(From OE-Core rev: e6d2b407979869219da1f15ed4b5c1c804548fce)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Started to run into this error on ubuntu 15.04
ERROR: Execution of event handler 'run_buildstats' failed
Traceback (most recent call last):
File "run_buildstats(e)", line 18, in run_buildstats(e=<bb.event.BuildStarted object at 0x7f93341c1b50>)
File "buildstats.bbclass", line 17, in set_device(e=<bb.event.BuildStarted object at 0x7f93341c1b50>)
SystemError: ../Objects/longobject.c:998: bad argument to internal function
ERROR: Command execution failed: Traceback (most recent call last):
File "/home/ubuntu/work/bleeding/openembedded-core/bitbake/lib/bb/command.py", line 102, in runAsyncCommand
commandmethod(self.cmds_async, self, options)
File "/home/ubuntu/work/bleeding/openembedded-core/bitbake/lib/bb/command.py", line 303, in buildTargets
command.cooker.buildTargets(pkgs_to_build, task)
File "/home/ubuntu/work/bleeding/openembedded-core/bitbake/lib/bb/cooker.py", line 1336, in buildTargets
bb.event.fire(bb.event.BuildStarted(buildname, fulltargetlist), self.data)
File "/home/ubuntu/work/bleeding/openembedded-core/bitbake/lib/bb/event.py", line 163, in fire
fire_class_handlers(event, d)
File "/home/ubuntu/work/bleeding/openembedded-core/bitbake/lib/bb/event.py", line 102, in fire_class_handlers
execute_handler(name, handler, event, d)
File "/home/ubuntu/work/bleeding/openembedded-core/bitbake/lib/bb/event.py", line 76, in execute_handler
ret = handler(event)
File "run_buildstats(e)", line 18, in run_buildstats
File "buildstats.bbclass", line 17, in set_device
SystemError: ../Objects/longobject.c:998: bad argument to internal function
similar to here
http://pycuda.2962900.n2.nabble.com/bad-argument-to-internal-function-td4063274.html
If it is right fix .. Dont know
Change-Id: I84f3a4043ad5246e080dfd7e2f066e5292d4af91
(From OE-Core rev: 5f85a7549d119177c5d39f965b949e957e07867e)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To avoid any further inconsistencies between buildstats and buildinfohelper
from toaster, buildstats will measure task duration using the time field
from within the TaskBase events: TaskStarted and TaskSucceeded/TaskFailed.
(From OE-Core rev: 406acd647a288694c2f776a9faa1f5607f3e8e7a)
Signed-off-by: Marius Avram <marius.avram@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These have been deprecated for a long time, convert the remaining
references to the correct modules and prepare for removal of the
compatibility support from bitbake.
(From OE-Core rev: 6a39835af2b2b3c7797fe05479341d71a3f3aaf6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that bitbake supports masking events for event handlers, lets use
this so event handlers are only called for events they care about. This
lets us simplify the code indentation a bit at least as well as mildly
improving the event handling performance.
(From OE-Core rev: bff73743280f9eafebe4591f7368ead91a4eb74d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This file is already opened a few lines previously, so drop the duplicated
call.
(From OE-Core rev: 4808629faa1222b31f92a3e410e06adb8e081293)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
python3 doesn't like files being left open. This updates the code style
to ensure file are closed.
(From OE-Core rev: ec74285bd2108f12f33fc8ac0dc1d124ab48be21)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It was pointed out we have a number of weird indentations in the python functions.
This patch cleans up 3, 7 and other weird indentations for the core bbclass files.
It also fixes some wierd (odd) shell function indentation which my searches picked up.
(From OE-Core rev: 8385d6d74624000d68814f4e3266d47bc8885942)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In OpenVZ containers (and probably lx containers as well),
the diskstats entry is not even present. Use the "NoLogicalDrive"
introduced by Elizabeth Flanagan in such case.
This allows the bitbaking to occure within such containers.
(From OE-Core rev: 16e09b850dcb44cb1afe411439e40a4bae7e8002)
Signed-off-by: Jean-François Dagenais <jeff.dagenais@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
tmpfs/encryptfs/(and most likely, but not confirmed)ramfs TMPDIRs
cause diskstats to choke. No device entry ends up in /proc/diskstats
for these fs types, which ends up causing the failure.
The short term solution is to exclude these fs types from diskstat
collection. Longer term we will want to see if we can collect
meaningful diskio for each of these, and other, use cases, but for
this cleans up Bug 1700.
[YOCTO #1700]
(From OE-Core rev: 2b14046c12855b6f484ba5bd6bc0a8022de6873e)
Signed-off-by: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
Corrected YOCTO bug location and format
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Task endtime got left out of recent changes. Adding it back in.
(From OE-Core rev: f1e4b049438218347922f13de810782be0205c3d)
Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit adds disk-io statistics functionality on a build
and per-task basis.
It pulls measurements for whatever partition TMPDIR exists on.
This data could be off if SSTATE_DIR and DL_DIR exist on
a different partition/volume.
Notes on what this pulls:
ReadsComp: Total number of reads complete
ReadsMerged: Total number of adjacent reads merged
SectRead: Total number of sectors read
TimeReads: Total number of m/s spent reading
WritesComp: Total number of writes completed
SectWrite: Total number of sectors written
TimeWrite: Total number of m/s spent writing
IOinProgress: Total amount of IO in progress at the time of
we look at /proc/diskstats.
TimeIO: Total number of m/s spent doing IO
WTimeIO: Weighted time doing I/O. From iostats.txt:
"This field is incremented at each I/O start, I/O completion, I/O
merge, or read of these stats by the number of I/Os in progress
(field 9) times the number of milliseconds spent doing I/O since the
last update of this field. This can provide an easy measure of both
I/O completion time and the backlog that may be accumulating."
(From OE-Core rev: eeebcebf0d695358e72f3aed753f66cddd5e0e61)
Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Pavel Modilaynen
Paul Eggleton
Joshua Lock
Patrick Ohly
Leonardo Sandoval
Mikko Ylinen
Stephano Cetola
Richard Purdie
Juro Bystricky
Randy MacLeod
Khem Raj
Marius Avram
Jean-François Dagenais
Elizabeth Flanagan