Commit Graph

72 Commits

Author SHA1 Message Date
Humberto Ibarra 5ff7db946c oeqa: fix typo
The word 'uninstall' was spelled incorrectly in various parts of
oeqa files.

(From OE-Core rev: af365e025030436f83b233fa51ecc9c58c58ce3c)

(From OE-Core rev: 9d25188e873645b849584b51a77b86588a51d4ba)

Signed-off-by: Humberto Ibarra <humberto.ibarra.lopez@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-23 12:49:48 -08:00
Aníbal Limón ba1aec3407 oeqa: Fix files handling on runtime tests.
Common files was move to oeqa/files from oeqa/runtime/files
because the same files are used across Runtime,SDK,eSDK tests.

(From OE-Core rev: f099302efe8f222c3e4ae3604429f5ede4fd8c67)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:19 +00:00
Aníbal Limón 14eee4fdf8 oeqa/sdkext: Adds case and context modules.
The extensible sdk context and case modules extends the sdk ones,
this means that the tests from sdk are run also the sdkext tests.

Enables support in context for use oe-test esdk command for run
the test suites, the same options of sdk are required for run esdk tests.

Removes old related to case and context inside oetest.py.

[YOCTO #10599]

(From OE-Core rev: 1f0bb99249744b87dd39227a4cf37f2341f5499c)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:19 +00:00
Aníbal Limón 8a37763ae9 oeqa/sdk: Add case and context modules for the SDK component
Adds case and context modules for SDK based on oetest.py old code.

Enables SDK Test component usage with oe-test, the SDK Test component
adds command line options for specify sdk installed dir, sdk environment
and target/hosts maniftest.

[YOCTO #10599]

(From OE-Core rev: 19e875dd81c42841666e6db5f6b665b4e1cddfe6)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:19 +00:00
Joshua Lock c4e2c59088 meta: remove True option to getVar calls
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.

Search made with the following regex: getVar ?\(( ?[^,()]*), True\)

(From OE-Core rev: 7c552996597faaee2fbee185b250c0ee30ea3b5f)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:23 +00:00
Ross Burton a7a6d05579 oeqa/oetest: show stderr when running commands
To help debug failures, redirect stderr to stdout in oeSDKTest.run() and
oeSDKExtTest.run().

(From OE-Core rev: 4cd143e0de7f0082f60f273f442f6255f28ec3e7)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-16 15:24:02 +01:00
Richard Purdie 896f1c7696 oeqa/oetest: Improve subprocess error reporting
Without this, we get to know the command failed and the exit code but
have no idea how the command failed since we don't get the output by
default.

This makes it much easier to see what went wrong and stand a chance of
fixing it.

(From OE-Core rev: b020b01d41ccaae5d679f1f7950af2e1a1788d39)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-17 10:23:43 +01:00
Mariano Lopez 65459f5b6d oeqa/oetest.py: Allow to export packages using symlinks
Currently packages that contains symlinks can't be extracted
and exported. This allows to export extracted such packages.

A nice side effect is improved readability.

[YOCTO #9932]

(From OE-Core rev: 0338f66c0d246c3b8d94ac68d60fbc4c314e500b)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-04 15:22:22 +01:00
Mariano Lopez 65a03c3f94 oetest.py: Remove check for __init__.py
When using "auto" in TEST_SUITES there is a check
for __init__.py in the <layer>/lib/oeqa/runtime/
directory in all the layers in BBLAYERS.

This check was needed because the way that python 2
import the modules, now that bitbake uses python 3
there is no need these __init__.py files, moreover
these files won't allow to import tests from other
layers.

This patch removes the check.

[YOCTO #9996]

(From OE-Core rev: f1cc272e4851fd994e9d052628a747ac19f90488)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-04 15:22:22 +01:00
Mariano Lopez 84d854c98b oetest.py: Add command line parameter support for tag in testexport
This allows to use a command line argument to change the
tag used to filter test instead of rebuilding the tests.

[YOCTO #8532]

(From OE-Core rev: 928e0eecdb126f7d0bacd05b7057fc825e0d8f05)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-04 15:22:22 +01:00
Mariano Lopez cd78d8745d oetest.py: Fix problem when there is more than one json file for package extraction
This allow to have more than one test to have json file in order
to install packages in the DUT without using a package manager.

[YOCTO #9926]

(From OE-Core rev: 129c2b4589710aa24e23108f2728f1625b72cb06)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-21 07:47:52 +01:00
Ross Burton 675843d540 oeqa: fix hasPackage, add hasPackageMatch
hasPackage() was looking for the string provided as an RE substring in the
manifest, which resulted in a large number of false positives (i.e. libgtkfoo
would match "gtk+").

Rewrite the manifest loader to parse the files into a proper data structure,
change hasPackage to do full string matches, and add hasPackageMatch which does
RE substring matches.

(From OE-Core rev: b9409863af71899e02275439949e3f4cdfaf2d0f)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-23 14:26:14 +01:00
Mariano Lopez 938f05333b oetest.py: Add install/uninstall functionality for DUTs
Add the functionality to install/unistall packages in the
DUTs without the use of the package manager. This is possible
with the extraction introduced in package manager class.

testimage and testexport bbclasses has been modified in order
to support this new feature.

[YOCTO #8694]

(From OE-Core rev: b7111d9e9d64d21f57729d1ac1865aea6e54cc8b)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-07 15:22:38 +01:00
Mariano Lopez d9b3ee8b38 lib/oeqa/otest.py: Fix import tests from other layers with python3
In python3 the functionality to import modules has been changed and
this broke the capability to add runtime tests from other layers.

This commit returns this capability to testimage and testexport.

[YOCTO #9705]

(From OE-Core rev: a26f23d3ce8f7e9f59dbc9bf27516377fd7a0a6d)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-07 15:22:38 +01:00
Richard Purdie 3b39971748 classes/lib: Complete transition to python3
This patch contains all the other misc pieces of the transition to
python3 which didn't make sense to be broken into individual patches.

(From OE-Core rev: fcd6b38bab8517d83e1ed48eef1bca9a9a190f57)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:01 +01:00
Richard Purdie a7309d5790 classes/lib: Update to use python3 command pipeline decoding
In python3, strings are unicode by default. We need to encode/decode
from command pipelines and other places where we interface with the
real world using the correct locales. This patch updates various
call sites to use the correct encoding/decodings.

(From OE-Core rev: bb4685af1bffe17b3aa92a6d21398f38a44ea874)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:00 +01:00
Mariano Lopez d56ccf691b oetest.py: Add support to copy unextracted packages for runtime testing
Sometimes is needed to have a package without extraction when
running a test. This patch adds the functionality.

[YOCTO #8536]

(From OE-Core rev: 49234fe926224c21ef6c8292132620b4716c5263)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-30 15:58:13 +01:00
Mariano Lopez 92afca7181 oetest.py: Add extract_packages() to RuntimeTestContext class
This new method extracts the content of package (RPM, DEB, or IPK)
to a directory inside of WORKDIR. The extraction is needed for later
install in the DUTs without using a package manager.

[YOCTO #8694]

(From OE-Core rev: 90d585f59f217f23694a9b02a73b79d18dfdb579)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-30 15:58:13 +01:00
Mariano Lopez 8a68910958 oetest.py: Add json file support to specify packages needed in runtime tests
This adds the functionality to use a json file to
specify the packages needed for a particular test.

The content of the json file is a dictionary with
dictionaries inside, using the test name as the hash.

The json file must have the same name as the class
module name and must be in the same path.

[YOCTO #7850]

(From OE-Core rev: 1f24ef9a339a2ad34e010f39aa93abdc8d085c85)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-30 15:58:13 +01:00
Mariano Lopez daf0cc3745 oetest.py: Move getTests() outside loadTests() method
The method getTests() can be useful to all the class,
not just to loadTests().

[YOCTO #8694]

(From OE-Core rev: 667a4549bd51a6e4b487006f4cc1542d2961757a)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-30 15:58:11 +01:00
Mariano Lopez b4c23e9e95 oetest.py: Use the real ExportTestContext in exported tests
There are parts of the fuctionallity missing when using the
dummy ExportTestContext class in runexported.py.

This changes the use of ExportTestContext dummy class from
runexported.py to the real class in oetest.py.

(From OE-Core rev: b0f105eecf25d334dd5fd0d936ec98cb8430ed23)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-19 09:05:18 +01:00
Mariano Lopez 86d65d79ad oetest.py: Add default pscmd to oeTest
pscmd is used by some tests to get the process
running on the target. If the test are exported
there won't be any pscmd attibute in the oeTest.

This adds "ps" as default pscmd.

(From OE-Core rev: 44aa8c3b6747179a0c4c156fc4922d838cdc19a4)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-19 09:05:18 +01:00
Mariano Lopez a9eb22e3d3 oeqa/oetest.py: Add class ExportTestContext
Adding the class is needed to separate the exported test
from the test image; both test run under different conditions,
i.e. an exported test doesn't require to change the signal
handling.

This change adds clasess ExportTestContext and ImageTestContext,
both of them inherits from RuntimeTestContext. Also refactors
RuntimeTestContext class, to keep the code common in this class.

(From OE-Core rev: ef9618a6e7f99e485cc6b59484d11807475c7823)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-19 09:05:17 +01:00
Ed Bartosh ecaf12ea5b oetest: make console output more verbose
Added output of test runner to bitbake console output.
bitbake <image> -c testimage now reports test progress
to the console:
NOTE: Executing RunQueue Tasks
...
NOTE: recipe core-image-lsb-1.0-r0: task do_testimage: Started
core-image-lsb-1.0-r0 do_testimage: test_ping (oeqa.runtime.ping.PingTest) ... ok
core-image-lsb-1.0-r0 do_testimage: test_ssh (oeqa.runtime.ssh.SshTest) ... ok
...
----------------------------------------------------------------------
core-image-lsb-1.0-r0 do_testimage: Ran 38 tests in 785.100s
core-image-lsb-1.0-r0 do_testimage:
core-image-lsb-1.0-r0 do_testimage: OK (skipped=1)

Adding more output to the console should also prevent autobuilder to kill
long running tests as ab assumes that test is stuck if no console output
produced by it for a long time.

[YOCTO #8238]

(From OE-Core rev: 06aa68138ec67623892221fbe9516bbb3795b48e)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-18 16:28:22 +01:00
Mariano Lopez d27ca36e2a oeqa/runexported.py: Fix exported test
With the changes introduced to test the eSDK
the runexported test failed during the execution.

This change fix runexported test in the least invasive
way, because of the release cycle.

(From OE-Core rev: da0abb9679cb1fd639859a2fdbd82101d0a81259)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-12 22:50:21 +01:00
Aníbal Limón 738bd1a640 classes/testsdk: Pass tcname to SDK and SDKExt contexts
tcname is needed for eSDK update testcase will be used for
 publish it and then try to update

(From OE-Core rev: a75944a63482597be88ff0f3ce55025647b78e2c)

Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:33:06 +00:00
Aníbal Limón 2a410b225a classes/testsdk: Move the removal of bitbake PATH to eSDK context only
The removal of bitbake and scripts PATH is only needed by eSDK tests
so move to eSDK context only.

This also it's a support for eSDK update test because it needs to
execute oe-publish-sdk from scripts.

(From OE-Core rev: 4ae0c84568f39661722cbceba8ddab22cffe5003)

Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28 11:33:06 +00:00
Aníbal Limón a619ea293a oeqa/oetest: Fix compatibility SDK tests using eSDK.
When run SDK tests over eSDK we need to use SDKExtTestContext
instead of SDKTestContext because if we use SDKTestContext search
for SDK manifest and depends on the SDK manifest generation so
populate_sdk needs to be executed.

Adds a compatibility mode flag to SDKExtTestContext for search tests
over sdk module instead of sdkext module and change testsdk calls
to comply with this new param.

(From OE-Core rev: ebe743235b383b17225553d84bf8e6f80d364dcd)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 16:06:23 +00:00
Aníbal Limón 062dbd6da7 classes/populate_sdk_ext: Add SDK_EXT_TARGET_MANIFEST and SDK_EXT_HOST_MANIFEST
Extensible SDK needs to point to the correct manifest so add
SDK_EXT_TARGET_MANIFEST and SDK_EXT_HOST_MANIFEST variables.

oeqa/oetest.py: Fix SDKExtTestContext for load the correct manifests.

(From OE-Core rev: 2310d5ad03531b7e1f9572c12c83c2fedc0291c9)

Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 16:06:23 +00:00
Aníbal Limón 4cfdf174ce testsdkext: Add skeleton for support Extensible SDK tests.
oeqa/sdkext: Add module and __init__.py will contain eSDK tests.
classes/testsdk: Add support for run eSDK tests.
oeqa/oetest: Create oeSDKExtTest for now only inherit oeSDKTest,
             modified SDKExtTestContext now inherit SDKTestContext
             and set sdkext filesdir for store data fixtures.

(From OE-Core rev: f3781544a5c077610498a6b7dc5244ee4c5bc6df)

Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 16:06:23 +00:00
Aníbal Limón 7181da7192 oeqa/oetest: oeSDKTest when run a command redirect env output to null
Some tests are failing because the eSDK env load script shows a help
message on the output so redirect this to /dev/null.

There was a discussion with Paul Eggleton [1], he don't want to have env
variables to change the behaviour of what the env script needs to print
so redirect the output in oeSDK.run() method. I didn't agree because
it can hides another messages like error ones and with the variable you
can control what prints or not.

[1] http://lists.openembedded.org/pipermail/openembedded-core/2016-February/116744.html

(From OE-Core rev: 7defb10d10861501947b4a686510e957c72313e6)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 16:06:23 +00:00
Aníbal Limón 3577c35f6e oetest.py/TestContext: Move loadTests and runTests inside it.
Method's for loadTests and runTests make sense to define
inside TestContext because it can be different around
Image, SDK, SDKExt.

(From OE-Core rev: 03af7b99e3ce36ce3e29dc31e33d2cc74eb14849)

Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 16:06:23 +00:00
Aníbal Limón 8009418d55 testimage/testsdk: Move get test suites routine inside TestContext.
In order to provide better abstraction move functions to get the test
suite inside the TestContext.

(From OE-Core rev: 552285688441330440bd1bc138aeaea87549024a)

Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 16:06:22 +00:00
Aníbal Limón b588b80d66 testimage/testsdk: Modularize TestContext.
Move anonymous duplicated class TestContext from testimage/testsdk to
oeqa/oetest now we have two new classes ImageTestContext and
SDKTestContext with common code in TestContext class.

(From OE-Core rev: 593f2fdf6ee94c5f91761a669048da3598cbe3fa)

Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 16:06:22 +00:00
Aníbal Limón 90590ab047 get_test_suites: Add sdkext type for load test suites.
(From OE-Core rev: 01a420dc9c0c04c793bbf5824db7c2b9d7c9de9b)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 16:06:22 +00:00
Aníbal Limón 3d1d30b0a3 testimage: Modularize helper functions for get test lists.
Test lists functions can be used in other parts so modularize it and
move to oeqa/oetest.py library.

Testimage class was updated to meet the new sign of the functions.

(From OE-Core rev: c9f771533af70e7ccb1e7064e58926cfaee7367a)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 16:06:22 +00:00
Mariano Lopez 3327401264 oetest: Add tearDownLocal class
In the current state there are some runtime test that
don't run the tearDown method fromm oeRuntimeTest class
because the tearDown class is overwritten in the child
class.

This change adds tearDownLocal method in oeRuntimeTest
class that will run after tearDown. This method can be
overwritten in the child classes to implement specific
test functionality.

[YOCTO #8465]

(From OE-Core rev: b5917f8032d6965596868b2fe01da4e0682e2804)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-19 17:57:59 +01:00
Mariano Lopez df09a6f4e3 oetest: Change logic of a failed test
Currently the logic to check if a test failed was
to check for an exception in the thread, but some
decorators used in the syslog runtime test would
generate and handle exceptions; this will mess
with the current check logic and will dump the
host and the target as if the test failed.

This patch changes the check logic to verify if
the test that just happend is in the failure test
list and dump the host and target accordingly.

[YOCTO #8406]

(From OE-Core rev: 8b97d9320b989023c62db8246f5d8d2126c3723e)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-01 07:43:34 +01:00
Richard Purdie 09af262045 oeqa/oetest: Fix SDK command execution
The SDK environment wasn't being sourced when running these test
commands, meaning in some cases the cross compiler was being tested,
not the SDK tools. This is clearly not the intent so fix this by
ensuring the SDK environment is present. This fixes test failures
in multilib SDKs.

(From OE-Core rev: 39c56f7c280cfe6bf40ea9145f3f7060b565bc62)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-28 12:00:18 +01:00
Lucian Musat f1e7fb0856 oeqa/oetest: Remove bb as requirement for oetest.
In order for the test export runner to work oetest needs to be
separated from bitbake environment. There is no need to use bb
import here so we can use a logger instead.

(From OE-Core rev: 22bd8a600823306d7c5965a9a69b8e8888993513)

Signed-off-by: Lucian Musat <george.l.musat@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-24 17:54:27 +01:00
Lucian Musat bbe06b443b oeqa/testimage: Enhance -v switch in testimage
When testimage is run with -v switch now individual
test progress can be seen directly in bitbake console.

[YOCTO #6841]

(From OE-Core rev: 27138b2eeafa8b81f3020ad9d8a55263e636288c)

Signed-off-by: Lucian Musat <george.l.musat@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-23 09:52:55 +01:00
Lucian Musat d8eb9d41b0 oeqa/decorators: Added decorator to restart the DUT in case of test hang.
Once the DUT is hanged during testing, currently all the following test
cases have to wait for default timeout to exit. Using this decorator the
user can choose a timeout at case by case basis and what happens when the
timeout is reached by overwriting the self.target.restart method.

[YOCTO #7853]

(From OE-Core rev: ce9a7501ea29ddba61ef7c297223b3f7eca5a2a1)

Signed-off-by: Lucian Musat <george.l.musat@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-18 09:05:21 +01:00
Mariano Lopez 6b41b7cc32 testimage: handle SIGTERM to conclude runqemu
In the current state if a SIGTERM is sent to
the testimage worker, the worker will exit but
runqemu and qemu won't exit and the processes
need to be killed manually to free the
bitbake lock.

This allows to catch the SIGTERM signal in
testimage, this way it is possible to stop
runqemu and qemu and allow to free the bitbake lock.

Also this allows to skip the rest of the tests
when running the tests in qemu or real hardware.

This also solves minimal breaks in the setup of the
runtime test when checking if qemu is alive.

[YOCTO #8239]

(From OE-Core rev: 2694d2f17d597b44fcc7aed5f6836081fa88a6b3)

Signed-off-by: Benjamin Esquivel <benjamin.esquivel@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-09 14:27:39 +01:00
Richard Purdie ef0fe3193e oeqa: Test failure/cleanup improvements
Currently, if qemu segfaults, the tests merrily continue trying to execute
which takes time for them to timeout and is a bit silly. Worse, no logs about
the segfault are shown to the user, its silent!

This patch tries to unravel the tangled web of issues and ensures that we:

* install a SIGCHLD handler which tells the user qemu exited
* check if qemu is running, if it isn't fail the test outright
* don't leave processes behind in sshcontrol which would hold
  bitbake.lock and block shutdown

(From OE-Core rev: 5c04b1ca1e989f569d5755a646734d01a0c56cae)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-06 15:26:18 +01:00
zjh fac0d67ec2 testimage: filter proper test cases by tags
If a test case is decorate by oeqa.utils.decorators.tag, this case will
by add a tag, testrunner will filter these tags by TEST_SUITES_TAGS
[YOCTO #7849]

(From OE-Core rev: 085589b1018ba4d950baf7bcfb499be02c1b29fc)

Signed-off-by: zjh <junhuix.zhang@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-04 16:23:52 +01:00
Mariano Lopez a2b0d04235 runexported: Make it compatible with host dump
Currently it is not possible to run a exported test,
but this patch will allow to use the HosDumper class
when running a exported test, otherwise the HostDumper
class will break runexpored test.

[YOCTO #8118]

(From OE-Core rev: 25814dbdd12db0499e4d81876873158e1b8a4fcc)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-03 12:43:17 +01:00
Mariano Lopez 53ab41a5f6 qemurunner: Added host dumps when there are errors
This adds an instance of HostDumper to qemurunner,
with this instance now is possible to get dumps
from the host when there is an error.

This adds dump points in the next cases:
    - runqemu exits before seeing qemu pid
    - Fail to get qemu process arguments
    - Not reach login banner before timeout
    - qemu pid never appears

This also modifies the constructors of BaseDumper,
HostDumper and TargetDumper, they don't require
the datastore anymore, but the feature to replace
datastore variables has been lost (never used)

[YOCTO #8118]

(From OE-Core rev: b0af40fb76cd5035696e9d8a44f815f64214d23a)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-03 12:43:16 +01:00
Mariano Lopez 277571bbd0 oetest: Fix regresion when testing real hardware
This fix the regresion introduced in commit
9c72c1a5aa
when testing with real hardware. This regression
happens when a test in real hardware fails.

[YOCTO #8203]

(From OE-Core rev: e63889cc70041ada022c2ebe789b569f9e44dbd6)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-30 12:36:13 +01:00
Mariano Lopez 85fd1a7a12 dump: Created new classes for dump host and target
It makes sense to separate the dump commands from the
oeRuntimeTest class, this way it can be used in all
the test context.

These are the changes included in this patch:

    - Created classes: BaseDumper, HostDumper, TargetDumper
    - Create an instance of HostDumper in imagetest.bbclass
      and add it to TestContext class, this way any class
      that have access to the TestContext would be able
      to dump logs from the host
    - Create an instance of TargetDumper in QemuTarget
      class after get the runner, this way it is
      accessible during the tests.

[YOCTO #8118]

(From OE-Core rev: ad10af6be343b5425fde43055263b0744c161cb3)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-30 12:36:12 +01:00
Mariano Lopez 48373be830 testimage: Run commands in target and host when test fails
This patch modify three files altought two of them
are minimal modifications. This version includes
the changes proposed by Paul.

testimage.bbclass:
    Create new vars for easy modification of the dump
directory and commands to be run on host and target
when a test fails
    TESTIMAGE_DUMP_DIR: Directory to save the dumps
    testimage_dump_target: Commands to run on target
    testimage_dump_host: Commands to run on host

oetest.py:
    - Allow to use the vars defined in testimage class
    - Now able to run commands in the host and dump the
      results
    - Fix an issue with the condition where to run the
      dump commands (Before it run the commands every
      test after a failure, now it runs the commands only
      in tests that failed)
    - Fix the output to stdout

[YOCTO #8118]

(From OE-Core rev: 26fe645457633f90bb5ddbb12f5f7b9ca4a06cc5)

(From OE-Core rev: 7b4fbbf979ed22434b8e3f83ae145139bb0d9fc7)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-30 12:36:07 +01:00