[YOCTO #8750] Allow oe-selftest to run custom test suites based on different criteria
1. Can run custom lists of tests based on different criteria:
--run-tests-by <name|class|module|id|tag> <list of tests|classes|modules|ids|tags>
eg: --run-tests-by module imagefeatures signing recipetool
--run-tests-by id 1377 1273 935
--run-tests-by tag wic sstate bitbake
2. Can list tests based on different criteria:
--list-tests-by <name|class|module|id|tag> <list of tests|classes|modules|ids|tags>
eg: --list-tests-by module imagefeatures signing recipetool
--list-tests-by id 1377 1273 935
--list-tests-by tag wic sstate bitbake
3. Can list all tags that have been set to test cases:
--list-tags
The list of tags should be kept as minimal as possible.
This helps preview the tags used so far.
To take advantage of the 'tag' feature:
- add @tag(feature=<>) to testcases
eg: @tag(feature='signing') for a single tag
@tag(feature=(('signing', 'sstate')) or
@tag(feature=['signing', 'sstate']) for multiple tags
(From OE-Core rev: 2d3a6d22e155911e39e4b7e323317f4a7cb1cb95)
Signed-off-by: Daniel Istrate <daniel.alexandrux.istrate@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Enable code coverage through the library 'python coverage'. In case the environment
variable COVERAGE_PROCESS_START is present (one of the requisites for measuring
sub-processes; the second one is including some coverage statements into the
python sitecustomize.py file) it will be taken into account, otherwise it is
exported with value '.coveragerc'. The latter value is a configuration file
(also automatically created) with some default settings. Once tests are
executed, a coverage report is shown on the log and the coverage output data is stored
with name '.coverage.<args>' where '<args>' is the name of the unit tests executed
or 'all_tests' when running with --run-all-tests. This output data can be latter used
for better reporting using the same tool (coverage).
As briefly indicate before, measuring sub-process implies setting the env variable
COVERAGE_PROCESS_START (done automatically by the oe-selftest code with this patch if
not already set) and creating a sitecustomize.py as explained on [1].
If either one of these is missing, complete coverage will be incomplete.
Current measurements for 'oe-selftest --run-all-tests' indicate that current coverage
is around 42 % taking into account BBLAYERS, bitbake and scripts folders. More details
on [2], indicating the coverage per file/module.
This tasks has been done together with Humberto Ibarra <humberto.ibarra.lopez@linux.intel.com>
[YOCTO #8679]
[1] http://coverage.readthedocs.org/en/latest/subprocess.html
[2] https://bugzilla.yoctoproject.org/attachment.cgi?id=2854
(From OE-Core rev: b3feee2cefbbd98b66dc395b651f47c5028c80a0)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In particular, this allows us to use code from bitbake's bb module
(such as tinfoil).
(From OE-Core rev: 5ca91cd0b56eb2da80c781fb46ae0b600c3f1eb9)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This ensures that oeqa.selftest.* from layers are found.
[YOCTO #7625]
(From OE-Core rev: 182b8ed9f26ed8b6a8eb6dcaec82db9aca14b010)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This ensures that all paths that hold selftest tests will be checked
(oeqa.selftest is a namespace package).
[YOCTO #7625]
(From OE-Core rev: 3c60cbced7b101ee52ce4a0a0bce542fd38f1821)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When oe-selftest starts it includes bblayers.inc into bblayers.conf
When oe-selftest ends it deletes bblayers.inc and the included line
from bblayers.conf
(From OE-Core rev: cf7bf27f565b34fdcd2caa25aaef068c0970965e)
Signed-off-by: Daniel Istrate <daniel.alexandrux.istrate@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The selftest can take a couple of hours to run, so add a custom result class to
timestamp the output to make it easy to spot any slow tests.
(From OE-Core rev: cce89c44948ee66ad0abb491be57e270038270e4)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
While trying to discover what tests are available, I felt the
need to be able to list all individual tests so I can run specific
tests.
This patch adds the "--list-classes" command line option that
lists the unit test classes and methods available.
(From OE-Core rev: f0ba400a398a172f9ebf953bb3a26e5d911a17d6)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixing a backtrace dump that happens if the script is started
without sourcing the oe-init-build-env first.
(From OE-Core rev: 0d769abcab272f41d74ed4d7915d26c7c309253a)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Regardless if the tests passed or not the script returned 0,
which isn't what one would expect.
(From OE-Core rev: c38f943c7fbb1fc077c875099dce8f73f41043b9)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The script should clean-up all the .inc files that might
have been created by tests regardless of the outcome or if
the script is interrupted. (currently the
last test will leave a conf/selftest.inc around, even
if it's not included anywhere)
Also fix delete_recipeinc to actually delete what's supposed to.
(From OE-Core rev: 6008745c56800e0f5f01a756be0701cebd9de4ae)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The purpose of oe-selftest is to run unittest modules added from meta/lib/oeqa/selftest,
which are tests against bitbake tools.
Right now the script it's useful for simple tests like:
- "bitbake --someoption, change some metadata, bitbake X, check something" type scenarios (PR service, error output, etc)
- or "bitbake-layers <...>" type scripts and yocto-bsp tools.
This commit also adds some helper modules that the tests will use and a base class.
Also, most of the tests will have a dependency on a meta-selftest layer
which contains specially modified recipes/bbappends/include files for the purpose of the tests.
The tests themselves will usually write to ".inc" files from the layer or in conf/selftest.inc
(which is added as an include in local.conf at the start and removed at the end)
It's a simple matter or sourcing the enviroment, adding the meta-selftest layer to bblayers.conf
and running: oe-selftest to get some results. It would finish faster if at least a core-image-minimal
was built before.
[ YOCTO #4740 ]
(From OE-Core rev: 41a4f8fb005328d3a631a9036ceb6dcf75754410)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Daniel Istrate
Leonardo Sandoval
Paul Eggleton
Christopher Larson
Ross Burton
Alexandru DAMIAN
Corneliu Stoicescu
Stefan Stanacar