In order to use the hasFeature and hasPackage methods, we need to make oeSDKTest extend oeTest and also set the test context (tc) attribute in the oeTest class when loading the tests.
(From OE-Core rev: 54436aeed5ac5e0da0779919f8524a0603e19c6a)
Signed-off-by: Corneliu Stoicescu <corneliux.stoicescu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- add support for sdk tests in the loadTests and runTests methods
- add new oeSDKTest test object
NOTE: Original patch made by: Richard Purdie <richard.purdie@linuxfoundation.org>
(From OE-Core rev: 6c147e5c82b39773e135ca828b37905cbb31be3b)
Signed-off-by: Corneliu Stoicescu <corneliux.stoicescu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to make the test skipping decorators independent of the oeTest object we rely on the unittest result object to construct skip, fail and error lists used by these decorators.
Created a new object getResults that analyses upper frames and retrieves the unittest result object instance, then return a list of failed, skipped and error tests.
Also removed the oetest import from decorators.py because it was no longer required.
(From OE-Core rev: 4d2d201158236bd4c72546cf8db88681ff921b11)
Signed-off-by: Lucian Musat <georgex.l.musat@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add the ability to export the tests so that they can run independently of
the build system, as is required if you want to be able to hand the test
execution off to a scheduler.
Booting/deployment of the target is still handled by the build system,
as before, only the execution of the tests happens outside of the build system.
Tests exported are the ones defined in TEST_SUITES.
No tests have been changed as interesting parts of the data store have been
exported and tests can continue to query them as before. Small adjustments were made
for a couple of oeqa modules though.
[YOCTO #5613]
(From OE-Core rev: 155dd52e0f707e06f50756584a50f744ba6b7844)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Restart is a property of a specific target, not
of a test class, should a test really need to restart
the target the direct method should be called.
Also some tests used this to enforce more ram, which
makes sense only for qemu targets only (and the inital
reason this was needed isn't valid anymore, qemu machines had
the default ram size bumped a while ago).
(From OE-Core rev: 333a4326082e500bdbcd323af37e183e74adf617)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch makes the necessary changes for using the targetcontrol.py module
so that one can run the same tests on a qemu instance or a remote machine
based on the value of TEST_TARGET variable: "qemu" or "simpleremote".
The default value is "qemu" which starts a qemu instance and it's the
with what we currently have.
With "simpleremote", the remote machine must be up with network and ssh
and you need to set TEST_TARGET_IP with the IP address of the remote machine
(it can still be a qemu instance that was manually started).
Basically testimage.bbclass now does something along the lines of:
- load tests -> deploy (prepare) / start target -> run tests.
There were a couple of changes necessary for tests and
also some cleanups/renames that were needed to adjust this change. (use
ip everywhere when refering to target and server_ip when refering to host/build machine)
Also two unnecessary and unsed methods were dropped from sshcontrol.
[ YOCTO #5554 ]
(From OE-Core rev: a7820350fa3271d78ed7476e02f4aef593be1125)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use the new manifest file instead of ${WORKDIR}/installed_pkgs.txt for determining
if an image has a certain package, because installed_pkgs.txt goes away with rm_work
enabled.
We can't use the IMAGE_MANIFEST var for the file path because that relies on IMAGE_NAME which
changes at every run (because of date), so we use the link which points to the last
one built.
[ YOCTO #5072 ]
(From OE-Core rev: f57c83fc33583c140f668946f3f3e79b960aa9ee)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add oeTest superclass, make oeRuntimeTest inherit that and
make the necesarry adjustments. Tests in lib/oeqa/runtime don't
change, they still inherit oeRuntimeTest.
We should do this because oetest.py in the future can be a base module
for more stuff than oeRuntimeTest.
(From OE-Core rev: cd4ed41a070bd52c446ac3df8337f17ab9421145)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Set longMessage to True for all tests derived from oeRuntimeTest, in
order to have somewhat info on assertions with cryptic or no messages.
(From OE-Core rev: e5196ebd6c652431a3b80cdc9f60b752401314f4)
Signed-off-by: Mihai Lindner <mihaix.lindner@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When a test module wants to be skipped because it doesn't
apply to the image but it was nevertheless a required
test (one in TEST_SUITES), we issued an warning that it
was a required test and went on with running the module.
Usually all tests in the module failed (e.g gcc tests on a non-sdk image),
but this allowed us to know that something went wrong with the image
(some package/feature didn't make it).
However, instead of just issuing an warning and running the tests
it's better to throw an exception. The traceback will tell us what's wrong,
and we don't run every single test method.
Output will look like this:
--snip--
| NOTE: Test modules ['oeqa.runtime.ping', 'oeqa.runtime.ssh', 'oeqa.runtime.gcc']
| NOTE: Found 5 tests
| test_ping (oeqa.runtime.ping.PingTest) ... ok
| test_ssh (oeqa.runtime.ssh.SshTest) ... ok
| ERROR
|
| ======================================================================
| ERROR: setUpModule (oeqa.runtime.gcc)
| ----------------------------------------------------------------------
| Traceback (most recent call last):
| File "/mnt/back/yocto/poky/meta/lib/oeqa/runtime/gcc.py", line 8, in setUpModule
| skipModule("Image doesn't have tools-sdk in IMAGE_FEATURES")
| File "/mnt/back/yocto/poky/meta/lib/oeqa/oetest.py", line 108, in skipModule
| "\nor the image really doesn't have the requred feature/package when it should." % (modname, reason))
| Exception:
| Test gcc wants to be skipped.
| Reason is: Image doesn't have tools-sdk in IMAGE_FEATURES
| Test was required in TEST_SUITES, so either the condition for skipping is wrong
| or the image really doesn't have the requred feature/package when it should.
|
| ----------------------------------------------------------------------
| Ran 2 tests in 1.036s
|
| FAILED (errors=1)
| NOTE: Sending SIGTERM to runqemu
--snip--
(From OE-Core rev: fd51cecf8b258d9f839a0ecebde69d09f75dc468)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tweak QemuRunner so we can actually restart the
qemu target in a test (if we want more memory for example).
Also add a restart method to the base test class so that tests
can use it.
(From OE-Core rev: 9de7fe11967576f4a8b24e653c6b9a02e5f6d85b)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds some comments to testimage.bbclass and the files it calls,
just to give an ideea of what it does.
(From OE-Core rev: 8bbb7116cf02466dfc59a17dc7bb51287aeea55b)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Many tests will use 'ps' but we need to know if it's busybox
or standard ps.
Drop the existing check from the connman test.
(From OE-Core rev: 1515d33d2c5b7275a3ac20e07c1db1d8273de796)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This module contains the base class for all runtime tests
and some helper methods.
(From OE-Core rev: 7765c27705e90381975fb2b89ea2181287517761)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Radu Moisan <radu.moisan@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Corneliu Stoicescu
Lucian Musat
Stefan Stanacar
Mihai Lindner