No need to do lsdir magic for finding buildstats when reading results.
(From OE-Core rev: 4502f0979bf2e8698bb196345b89b170641fd43f)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of archiving buildstats in raw text file format convert all
buildstats into one json-formatted file. Some redundant information,
i.e. 'Event:', 'utime:', 'stime:', 'cutime:' and 'cstime:' fields, are
dropped.
(From OE-Core rev: efcf74b194f2a40eb3e6359dd41386db3eb25287)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add data from /proc/<pid>/io to system resource measurements.
(From OE-Core rev: e69a46a77854fac1169a09e0c5b70fa4b972255a)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use Python standard library functionality instead of the time utility
for measuring elapsed (wall clock) time of commands. The time.* log
files are also ditched. However, the same detailed resource usage data,
previously found in time.* logs is now provided in results.json file.
This data is collected through the resource module of Python.
(From OE-Core rev: d5ad818dd501b18379aa3540bffa9b920d7c3bab)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Change directory name from 'buildstats-<test_name>' to just
'buildstats'. However, this patch adds the possibility to label
buildstats directory name with a postfix which makes it possible to save
multiple buildstats per test, for example.
(From OE-Core rev: 8997556040b2e7bfcfa6a75d4d97eb2e32207217)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Store the output data of each test in an individual subdirectory instead
of storing everything in the root output directory.
(From OE-Core rev: 64ff34df96aa9a74dd4303f76ec711aa5e9d5030)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Archive buildstats in a directory like 'buildstats' instead of something
like 'buildstats/20160513120000'.
(From OE-Core rev: 95138cdc70bb7f9b7ab74e1d83305f009790dccc)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Automatically create a json.formatted file (results.json) in the results
directory that contains results from all tests.
(From OE-Core rev: 6df3263531a41805b2280bb999cb4a73f9f91eae)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This defaults to 'oe-core' but can be defined using the
OE_BUILDPERF_PRODUCT environment variable.
(From OE-Core rev: a22cc3e04001be5d11bd85dbdceb7088cae7c735)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now failed measurements correctly cause a test failure (recorded as an
error). There should be no need to continue the test if one step fails,
especially now that the tests don't depend on each other.
(From OE-Core rev: 446e32aadc775ca146d12173b1463f524d7fe6ef)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add test set-up functionality so that the individual tests do not depend
on each other. This should make sure that a failure in one test does not
affect the results of another test. The patch also makes it reasonable
to run only a subset of the tests by using the --run-tests option.
The increase in total execution time of the full suite - caused by the
additional set-up steps - is insignificant because normally no
additional tasks need to be run. The previous test has already done all
set-up work.
(From OE-Core rev: 69b3c63e32d09ea4a41b21daacdff6bf1fc447c1)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The test status check done when writing globalres log was incorrect.
(From OE-Core rev: 3efbd49fd80d2b349a8fd44dbcd509168dbc1061)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Makes it possible to create easily sortable tags. Also, the default tag
format is updated to use the new keyword.
(From OE-Core rev: e3161654d75dfc3b059c519205b38b26e3ffb215)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This number represents the number of commits since the beginning of git
history until the tested revision. This helps e.g. in ordering results.
(From OE-Core rev: b52070dd057ff5b410cd193f9be2f25bc4c506cc)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This makes it possible to create numbered tags, where the "basename" of
the tag is the same and the only difference is an (automatically)
increasing index number. This is useful if you do multiple test runs on
the same commit. For example, using:
--commit-results-tag {tester_host}/{git_commit}/{tag_num}
would give you tags something like:
myhost/decb3119dffd3fd38b800bebc1e510f9217a152e/0
myhost/decb3119dffd3fd38b800bebc1e510f9217a152e/1
...
The default tag format is updated to use this new keyword in order to
prevent unintentional tag name clashes.
(From OE-Core rev: cf2aba16338a147f81802f48d2e24a96c7133548)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create a Git tag when committing results to a Git repository. This patch
also implements --commit-results-tag command line option for controlling
the tag name. The value
is a format string where the following fields may be used:
- {git_branch} - target branch being tested
- {git_commit} - target commit being tested
- {tester_host} - hostname of the tester machine
Tagging can be disabled by giving an empty string to
--commit-results-tag. The option has no effect if --commit-results is
not defined.
(From OE-Core rev: 60059ff5b81d6ba9ba344161d51d1290559ac2df)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A new command line option for defining the branch where results are
commited. The value is actually a format string accepting two field
names:
- {git_branch} expands to the name of the target branch being tested
- {tester_host} expands to the hostname of the tester machine
The option has no effect if --commit-results is not used.
(From OE-Core rev: b54b63395ec632748a57a702812c8a9a07af35ab)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Implement a new command line option '--commit-results' which commits the
test results data into a Git repository. The given path must be an
existing initialized local Git repository.
(From OE-Core rev: b6f635513ca971402e7a970acc2168fb5d4a9476)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is basically a internal change, at this point. Term 'commit' better
represents the data we actually have. Term 'revision' is more vague and
could be understood to point to a tag object, for example.
(From OE-Core rev: f49cf7959b8aaa52b79b22a5884c6aa580a50302)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix a bug that was introduced when converting to unittest framework.
(From OE-Core rev: 3bdb7b2e512b2f160360e95ed5b2be3871ec0b4b)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Drop the self-baked force_rm() method.
(From OE-Core rev: c86bf80abd87acb0da5860806822c64ec9dee089)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Log failures of commands whose output is stored.
(From OE-Core rev: 240f6e7366c8a9ea830e531d307dd2e27a61a6bd)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Special runCmd() for build perf tests which doesn't raise an
AssertionError when the command fails. This causes command failures to
be detected as test errors instead of test failures. This way "failed"
state of tests is reserved for future making it possible to set e.g.
thresholds for certain measurement results.
(From OE-Core rev: 09590ac76a19ee1b1b4a9188f7fce5029f0de52a)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit converts the actual tests to be compatible with the new
Python unittest based framework.
(From OE-Core rev: 4e81967131863df7ee6c8356cb41be51f1b8c260)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The new class is derived from unittest.TextTestResult class. It is
actually implemented by modifying the old BuildPerfTestRunner class
which, in turn, is replaced by a totally new simple implementation
derived from unittest.TestRunner.
(From OE-Core rev: 89eb37ef1ef8d5deb87fd55c9ea7b2cfa2681b07)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rename BuildPerfTest to BuildPerfTestCase and convert it to be derived
from TestCase class from the unittest framework of the Python standard
library. This doesn't work with our existing testcases or test runner
class and these need to be modified, too.
(From OE-Core rev: b0b434210a3dbd576f68344e29b8c20d18561099)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add logic for the cases when the time retrieved does
not have decimal part.
(From OE-Core rev: a6c9e515f8bc590612e3082ab1c4c254711c8e3b)
Signed-off-by: Jose Perez Carranza <jose.perez.carranza@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using this option the script appends test results into a 'global results
file'. A CSV-formatted output of the results. This option is to provide
compatibility with the old build-perf-test.sh.
(From OE-Core rev: e9f18e63220e452f2b0c878998e57d944ae83980)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BuildPerfTestRunner determines these from the Git repository under which
it is being run (i.e. where the build directory exists). The branch and
revision may be defined/overridden with OE_BUILDPERFTEST_GIT_BRANCH
and OE_BUILDPERFTEST_GIT_BRANCH environment variables, if needed. This
makes it possible to run the build performance test script even if the
top directory is not a git repository clone, for example.
(From OE-Core rev: e6004582454d8c6a18f617c12e6e408ded5be8df)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Have the build/conf directory as part of test results.
(From OE-Core rev: ea32a8a65f668450e0ec0e80483bd5e8c91b8a18)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Re-implement "test4" from build-perf-test.sh which measures eSDK metrics.
(From OE-Core rev: c449ca1b062426380a0572aed2da4ee4a83281f5)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Re-implement "test3" from build-perf-test.sh which measures
bitbake parsing time.
(From OE-Core rev: 388c7160b90247df9c78d8803083821402c13af5)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Re-implement "test1_p3" from build-perf-test.sh which measures
'bitbake core-image-sato' with rm_work enabled.
(From OE-Core rev: bfe48d343e8a281760da0b6744c08d47039f93e0)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Re-implement "test1_p2" from build-perf-test.sh which measures
'bitbake virtual/kernel'.
(From OE-Core rev: 5ff82c6e2b0ff41a9bfdd49c99e84a982cf6e467)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Re-implement "test1_p1" from build-perf-test.sh which measures
'bitbake core-image-sato'.
(From OE-Core rev: 1ddd4a8ce4e2053def0e946d4e4b4b97bdc28e61)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The new class is responsible for actually running the tests and
processing their results. This commit also adds a decorator function for
adding new tests. No automatic test discovery, at least yet.
(From OE-Core rev: bf90aecb7e150d6bfac7240286c797b79d26528b)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a new method to BuildPerfTest class for measuring the disk usage of
a file of directory.
(From OE-Core rev: 85cdc240e75d481e93238fbf75f8b8431da05f19)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add new methods to BuildPerfTest class for running a shell
command and logging its output.
(From OE-Core rev: 8f0b11ba1266f9c650cf34d9b394d72009ee7207)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Extend BuildPerfTest class with a new method for measuring the system
resource usage of a shell command to BuildPerfTest class. For now,
easurement of the elapsed time is done with the Gnu time utility,
similarly to the build-perf-test.sh shell script. And, it currently only
records the elapsed (wall clock).
The measured values (currently, only the elapsed time) is actually a
dictionary, making it possible to extend it with additional resource
values, e.g. cpu time or i/o usage, in the future. In addition to the
actual values of the measurement each record contains a 'name' and
'legend' where name is supposed to function as a common key or id over
test runs, making comparison and trending easier, for example. Legend is
supposed to be a short human readable description.
(From OE-Core rev: ced156bfea4a6649d201f41275641a633f218322)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The new class will be used as an abstract base class for build
performance tests. This implementation contains some common
functionality used in multiple tests, "copied" from the
build-perf-test.sh shell script.
(From OE-Core rev: 35cd7363759a286e80ddca0d028db3d2bf524b17)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a new utility class for dropping Linux kernel caches. It uses sudo
and tee to write to the drop_caches file. Checking if the user has the
permissions to drop caches (without a password) is done by trying to
writing an invalid value to the drop_caches file. This way, we will find
if writing (with tee) is possible but not really dropping caches, yet.
User can avoid giving the password by adding something like:
<user> ALL = NOPASSWD: /usr/bin/tee /proc/sys/vm/drop_caches
to the system sudoers file.
(From OE-Core rev: c9cb248429ced50c96d11ba5361c272d4c9b9323)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Wireframe of a new Python module for containing build performance tests
and utility functions.
(From OE-Core rev: b44af9051a9da8d2cddc7b071a896331967f15cc)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Markus Lehtonen
Jose Perez Carranza