Current targets (ssh and qemu) require a logger in their
constructors, so in order to get a new target we need
to provide the logger.
[YOCTO #10686]
(From OE-Core rev: 26bae3c255bc1e1cc8d81db0cffc809de0182a43)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit will prepare context to use qemu and not just
simpleremote.
(From OE-Core rev: 7485a08c967916fb6edff4cc573d9314ec577031)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This add the option to select what tests to run in the
<module>[.<class>[.<test>]] format.
Currently it just support modules
(From OE-Core rev: 0050565d71cee5a0cc22660b2398e4334d4b6719)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
skipIfNotDataVar will skip a test if a variable doesn't have certain value.
skipIfNotInDataVar will skip a test if a value is not in a certain variable.
(From OE-Core rev: a81045f4e2b740173237f5ae4e80e2bc0b287faa)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As part of the refactor we require to move the files used
in runtime testing to the new directory. This also adds
the path to the runtime test context.
[YOCTO #10234]
(From OE-Core rev: f80c21b46f69c42ff0853bf8abd01cf0c082c346)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds a new decorator to check if image under tests has
certain DISTRO_FEATURE or IMAGE_FEATURE.
[YOCTO #10234]
(From OE-Core rev: 8740803d0696a0e97b72210a56f4fbd3135826ed)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This new decorator will be used to skip the test
if the image under test doesn't have the required
packages installed.
[YOCTO #10234]
(From OE-Core rev: 021449938ff0b4d182d7f02930a80693f109c8ba)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds OERuntimeTestCase, OERuntimeTestContext, and OERuntimeTestLoader
to be used for runtime testing.
As expected there are some changes in runtime context:
- Adds the target to be used for runtime testing, the default
is a SSH connection to the device under test running a OE image.
- Runtime context requires image manifest because several
tests are skipped if a package is missing or installed.
- Several tests require the output of the ps command and it changes
its output and arguments if busybox o procps is installed, so the
case must use the correct ps command.
[YOCTO #10234]
(From OE-Core rev: f995f178de79d6d11422cd879d06371811f50651)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With this commit now it is possible to add targets with SSH for testing.
Most of it was imported for existing code, with improvements in log
handling.
[YOCTO #10234]
(From OE-Core rev: 3bc13548df4adb85f09467d200530a9c9f60da04)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The new oeqa core framework will modify the structure of the runtime
folder the new runtime folder will have python code inside to support
runtime test cases.
(From OE-Core rev: 637b712096e9d230e15b1a432a561e4118db34c8)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Common files was move to oeqa/files from oeqa/runtime/files
because the same files are used across Runtime,SDK,eSDK tests.
(From OE-Core rev: f099302efe8f222c3e4ae3604429f5ede4fd8c67)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
TargetBuildProject was refactored to avoid bitbake dependency so
the instance don't allow to pass data store anymore.
classes/testimage: Export proxies before run tests
The TargetBuildProject based tests download archives from network.
(From OE-Core rev: e275f29de500a338a02402ecc570405309963b35)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Summary,
- Changes base case class to OESDKExtTest.
- Changes decorator classes to new ones.
- Chnages variable names sdktestdir -> sdk_dir.
- Added missing license to MIT.
(From OE-Core rev: 49568055df0a64e4228f27130b13ccafbba2a460)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The extensible sdk context and case modules extends the sdk ones,
this means that the tests from sdk are run also the sdkext tests.
Enables support in context for use oe-test esdk command for run
the test suites, the same options of sdk are required for run esdk tests.
Removes old related to case and context inside oetest.py.
[YOCTO #10599]
(From OE-Core rev: 1f0bb99249744b87dd39227a4cf37f2341f5499c)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For match with the new structure of the OEQA framework.
In the new framework Test component base directory in this case
sdk module will contain case and context implementations.
[YOCTO #10599]
(From OE-Core rev: bdb92fa4d9bd2e4a0a14e3adc62a6b9e9bf639d3)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The TEST_LOG_DIR was used for store sdk_target_log this log
contains the output of the run of build commands now that information
could be found also on log.do_testsdk under WORKDIR.
The log will continue to store into SDK_DIR instead of TEST_LOG_DIR.
(From OE-Core rev: 1c9ba4b698bab916d42b58255692a7bf3d773bbc)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Summary of the changes:
- Remove auto extend_path using pkgutil at __init__, is not needed.
- Change base class to OESDKTestCase.
- Add td_vars attr to set dependencies of certain variables in test
data.
- Change skips from module level to class level because Test context
(tc)
now isn't at module level.
- Variable names changes to be consistent (i.e. sdktestdir ->
sdk_dir).
[YOCTO #10599]
- Don't use bb.utils functions use instead remove_safe and shutil
for copy files.
- SDKBuildProject pass test data variables instead of call getVar
inside.
[YOCTO #10231]
(From OE-Core rev: 91cd1ed19a3f34c29cd77eb136036975fe465444)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Don't use bitbake references inside utils modules, in order todo
that changes getVar calls for arguments in the __init__ method like
dl_dir for all the classes and testlogdir, builddatetime in
SDKBUildProject.
Also don't export proxies inside _download_archive method, a good
practice is to setup the proxies at init of the process instead of
do it in this helper module.
[YOCTO #10231]
[YOCTO #10599]
(From OE-Core rev: 581c34d1efe9839f50ef322761269b4e4d8a56a6)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The new buildproject module will contain only BuildProject class
a helper class for build source code.
The remaining classes TargetBuildProject and SDKBuildProject was
move to runtime and sdk respectively.
[YOCTO #10599]
(From OE-Core rev: 525fd2a5cda00890e921b63f7f608a10bc024d73)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The functionality provided is the same with imporvements on code
reuse and better interfaces.
(From OE-Core rev: 7a1ae3149965b162fb2c71fc7067e07a7a189249)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds case and context modules for SDK based on oetest.py old code.
Enables SDK Test component usage with oe-test, the SDK Test component
adds command line options for specify sdk installed dir, sdk environment
and target/hosts maniftest.
[YOCTO #10599]
(From OE-Core rev: 19e875dd81c42841666e6db5f6b665b4e1cddfe6)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It doesn't make sense to have files related to sdk module into runtime
module.
[YOCTO #10599]
(From OE-Core rev: 9a3be58b17afbe4ef00030b0e6ad8b20b03adc49)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For match with the new structure of the OEQA framework.
In the new framework Test component base directory in this case
sdk module will contain case and context implementations.
[YOCTO #10599]
(From OE-Core rev: 57af8ee4021c302bd351adf03e6d85274ad7efd5)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Those files are used by runtime and sdk test cases, so move to
base directory of oeqa module.
[YOCTO #10599]
(From OE-Core rev: ec73e8a3d3149f3866b7bfc06f169c6e05e2d338)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bitbake logger changes the way debug is logged and adds
different levels within debug, this is passed as argument
to the function and breaks compatibility with vanilla loggers.
This implements a way to handle this adding a new function for
debug, that will dispatch the correct logging method signature.
Also overrides info method to use logging.INFO + 1 in order to
see plain data.
Also this commit fix the issue of not showing the test summary
and results when running from bitbake.
[YOCTO #10686]
(From OE-Core rev: 619c9ab308fbef9e3563dc661e432603e764b562)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This function will generates testdata.json per SDK type.
[YOCTO #10231]
(From OE-Core rev: bff1b66c6b88fd2d99d14cecba27ee2f7462711f)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This function will generates testdata.json by image type.
[YOCTO #10231]
(From OE-Core rev: 4c0061422df472395ca0caa76ef92adb860f1167)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The export2json function export the variables contained in
the data store to JSON format, the main usage for now will be
to provide test data to QA framework.
(From OE-Core rev: 57c7bf68ed66a56601e1431bb2db750c5742b5ce)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The README has an introduction and explains how to run the test suite
and creates a new Test component.
(From OE-Core rev: 9d474172c47695be1a61538f5b87ca8d9db25fa7)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Serves as an first input of how to the OEQA framework works.
(From OE-Core rev: 115f80adf1b230c5d0392e7833e9aeb274642bcb)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The OETestContextExecutor class supports to use oe-test for run core
test component also is a base class for the other test components
(runtime, sdk, selftest).
Te principal functionality is to support cmdline parsing and execution
of OETestContext, the test components could extend the common options
to provide specific ones. The common options between test components
are test data file, output log and test cases path's to scan.
Also it initializes the logger to be passed to the whole OEQA framework.
[YOCTO #10230]
(From OE-Core rev: 039deafa5f2c8fab31b8373b39f8bc219377b893)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The new oe-test script will be use to run test components with
one single script.
The oe-test script search for test components inside meta/lib/oeqa,
the test components needs to implement OETestContextExecutor inside
context module in order to be supported by oe-test.
[YOCTO #10230]
(From OE-Core rev: 04b69cff3957731fa1ed2f7d23f2f616978ed0b7)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This test suite covers the current functionality for the OEQA
framework.
For run certain test suite,
$ cd meta/lib/oeqa/core/tests
$ ./test_data.py
(From OE-Core rev: 7d7d0dc3736fc12ae7848de2785f0066e6470cd1)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The OETestDataDepends decorator skips a test case if a variable
isn't into test data (d).
The skipIfDataVar decorator skips a test case if a variable
has certain value.
(From OE-Core rev: 7dc519d20e835ee7693c31903e164c4bc0e5e598)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The OETimeout provides support for specify certain timeout
in seconds for a test case, if the timeout is reach the SIGALRM
is sent and an exception is raised to notify the timeout.
[YOCTO #10235]
(From OE-Core rev: 1bf66a370361912e9950d7ff45e382c93622a169)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These two decorators stores certain TAG or ID for the test case
also provides support for filtering in loading step.
[YOCTO #10236]
(From OE-Core rev: 047af4ce864bbf98e2617b348ae9ccb77ac52871)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The OETestDepends decorator could be used over test cases to
define some dependency between them.
At loading time sorting the tests to grauntee that a test case
executes before also raise an exception if found a circular
dependency between test cases.
At before test case run reviews if the dependency if meet, in the
case of don't it skips the test case run.
(From OE-Core rev: 2385bd3c8a7c012fd1cad5465ec7d34675552c75)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
loader: Implements OETestLoader handling OETestDecorator
and filtering support when load tests. The OETestLoader is
responsible to set custom methods, attrs of the OEQA
frameowork.
[YOCTO #10231]
[YOCTO #10317]
[YOCTO #10353]
decorator: Add base class OETestDecorator to provide a common
way to define decorators to be used over OETestCase's, every
decorator has a method to be called when loading tests and
before test execution starts. Special decorators could be
implemented for filter tests on loading phase.
context: Provides HIGH level API for loadTests and runTests
of certain test component (i.e. runtime, sdk, selftest).
[YOCTO #10230]
(From OE-Core rev: 275ef03b77ef5f23b75cb01c55206d1ab0261342)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
misc: Functions for transform object to other types.
path: Functions for path handling.
test: Functions for operations related to test cases and suites.
[YOCTO #10232]
(From OE-Core rev: 102d04ccca3ca89d41b76a8c44e0ca0f436b7004)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
case: Defines OETestCase base class that provides custom
methods/attrs defined by the framework.
Every OETestCase instance contains a reference to the test
data (d), the test context (tc) and the logger.
Also implements _oe{SetUp,TearDown}Class for make special
handling of OEQA decorators and validations.
runner: Defines OETestRunner/OETestResult with support for RAW
and XML result logs.
exception: Custom exceptions related to the OEQA framework based
on class OEQAException.
[YOCTO #10230]
[YOCTO #10233]
(From OE-Core rev: c466086ccc4d4bb02d578a821cfb945945bfd529)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Used wic-tools target to get location of syslinux.
(From OE-Core rev: 5afd7def3b101bb3d650db4d005cb8d36aaf074a)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rebuild wic-tools if its native sysroot doesn't exist to ensure
that all required tools are available.
(From OE-Core rev: 0d005d099a2b8ee1303b98710cdc78e06e14ab39)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This parameter will be used to allow or prevent variable caching.
For example, we don't want to cache value of RECIPE_SYSROOT_NATIVE
wic-tools variable as we may decide to rebuild wic-tools.
(From OE-Core rev: e4269fdb4c3ef06b97df063f8586f74986215c83)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed test_systemd_bootdisk, test_wic_image_type and test_qemu
test cases by building core-image-minimal with correct configuration.
(From OE-Core rev: 897fe85d34302953c98d07ade4fa2dd749ae2d22)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Setting WKS_FILE variable should be done only when
wic image is expected to be built by bitbake.
If it's set for all images it breaks image building in
some cases.
(From OE-Core rev: 46b84310f7df157c2ef290a60f5c2136d4206f09)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Mariano Lopez
Aníbal Limón
Ed Bartosh