The purpose of this module is to deploy a test image on a EFI-enabled hardware
and run our runtime tests. A bit of background:
- testimage.bbclass uses the concept of TEST_TARGET which is a class name
that is responsible for target deploying. A layer can provide
it's own TEST_TARGET. Right now has OE-core has a QemuTarget and a SimpleRemoteTarget
(ssh into an already up and running machine and run tests), the default one being qemu.
- basically testimage does something like:
target.deploy()
try:
target.start()
runTests()
finally:
target.stop()
This module assumes a running EFI machine with gummiboot as bootloader and
core-image-testmaster installed (or similar). Also your hardware under test has
to be in a DHCP-enabled network that gives it the same IP for each reboot.
One time setup (master image):
- build core-image-testmaster with EFI_PROVIDER = "gummiboot"
- install the image on the target
Test image setup:
- build your test image, e.g core-image-sato as you usually do, but with these in local.conf:
IMAGE_FSTYPES += "tar.gz"
- Now run the tests:
INHERIT += "testimage"
TEST_TARGET = "GummibootTarget"
TEST_TARGET_IP = "192.168.2.3"
bitbake core-image-sato -c testimage
Other notes:
- TEST_POWERCONTROL_CMD (togheter with TEST_POWERCONTROL_EXTRA_ARGS) can be a command that runs on the host and does power cycling.
The test code passes one argument to that command: off, on or cycle (off then on). In my case I use something like
TEST_POWERCONTROL_CMD="powercontrol.exp test 10.11.12.1 nuc1" in local.conf.
Basically my expect script does: 'ssh test@10.11.12.1 "pyctl nuc1 <arg>" and runs a python script there that controls power for a label called nuc1'.
The reason why my expect script has to ssh into another machine is because of network topology, and that machine is the one actually connected
to the test rack and the power strip. That's why TEST_POWERCONTROL_CMD and _ARGS need to be customized for one's setup, the only requirement being
that it accepts: on/off/cycle as the last argument.
- if no command is defined it would use classic reboot. This is fine as long as the machine
actually reboots (as in the ssh test hasn't failed), but it's useful for "simple-setup-with-one-board-on-the-desk" scenario, where
some manual interaction is okay from time to time.
[YOCTO #5614]
(From OE-Core rev: e00f888a88d0851b088c232dec66418e575a2e90)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We see these from qemu images on servers under load. They're not interesting
and clutter the build output so lets ignore them.
(From OE-Core rev: 053eba4885ab4016f7efc7899ec9e02df3384d11)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
TEST_SUITES = "auto" would run smart tests even for non-rpm images,
which is pointless.
(From OE-Core rev: 1913f38d79a56f7d21999b6839835f967ca28abd)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A previous commit broke downloads
when proxies are involved, let's fix it.
(From OE-Core rev: 97e263b99cbe8184a74f80738fd471cfdef29e0c)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The settle() function had a race where services could still be activating at two
minutes but then when the final log is output, they've activated.
Remove this race and generally clean up the code.
(From OE-Core rev: 8d107e0a828868702cfe035104c1f0b51da4291e)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this it won't add core2-64 and similar channels, as the
directory name in deploy_dir/rpm uses _ not - as the package arch.
(From OE-Core rev: 14e3a8f58e309121760fec70619633281dd9d88c)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Restart is a property of a specific target, not
of a test class, should a test really need to restart
the target the direct method should be called.
Also some tests used this to enforce more ram, which
makes sense only for qemu targets only (and the inital
reason this was needed isn't valid anymore, qemu machines had
the default ram size bumped a while ago).
(From OE-Core rev: 333a4326082e500bdbcd323af37e183e74adf617)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Translate TUNE_PKGARCH to find the right file (this
used to work because tune for qemux86-64 used to be
x86_64 now it's core2-64)
Also, while using packagedata was nice, it's harder to make
the test exportable and runnable outside of the
build system. (where oe.packagedata isn't available)
(From OE-Core rev: fd0e9ad4d295ca11b33c3e3e11069421dee834e8)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When checking that no services have failed to start, actually wait for services
to finish starting by waiting for there not be no units in the "activating"
state.
(From OE-Core rev: 4d6422a84eba005a6fd788ce18c9dd42b079e2a8)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Run a python script on the target
1)checks the output.
2)Call os.system method create a testfile
(From OE-Core rev: 4465c9368b0c37a3a2c41b68f65de08690a8179b)
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
They are basically the same tests but:
- they look cleaner, using one single method / assert
- output from unittest will be cleaner (and includes a verbose status when needed)
- they are better grouped and use a real, active, enabled service
(machineid will be dropped and hostnamed was a static service)
(From OE-Core rev: 0aa5b97f9e0d8941ce342e9f162dd3b19d4b49bb)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the hostnamed service can't be started or stopped, show the output from
systemctl status to assist debugging.
(From OE-Core rev: 024cc1cee3a9954272ecbbff1ba7153725c56dce)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If some services have failed to start, get the status of them and some of their
log to help debug the problem.
(From OE-Core rev: e371765431be25b81dbdb385233e3db5851e59d0)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If connman isn't running and we're running under systemd, use systemctl to get
the state according to systemd and the end of the connman log.
(From OE-Core rev: e77bcc8e4baf11701b105d071a979a21a2a303d5)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch makes the necessary changes for using the targetcontrol.py module
so that one can run the same tests on a qemu instance or a remote machine
based on the value of TEST_TARGET variable: "qemu" or "simpleremote".
The default value is "qemu" which starts a qemu instance and it's the
with what we currently have.
With "simpleremote", the remote machine must be up with network and ssh
and you need to set TEST_TARGET_IP with the IP address of the remote machine
(it can still be a qemu instance that was manually started).
Basically testimage.bbclass now does something along the lines of:
- load tests -> deploy (prepare) / start target -> run tests.
There were a couple of changes necessary for tests and
also some cleanups/renames that were needed to adjust this change. (use
ip everywhere when refering to target and server_ip when refering to host/build machine)
Also two unnecessary and unsed methods were dropped from sshcontrol.
[ YOCTO #5554 ]
(From OE-Core rev: a7820350fa3271d78ed7476e02f4aef593be1125)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The connman unique test starts another instance of connmand and then does a ps to
verify that there's only one of these running, on the assumption that the new
one has quit because there's already one running (started by init).
However, connmand is forking into the background straight away so there's a race
between running ps and the second connmand discovering the first and exiting.
This race can be seen because the test displays the output of ps, and by the
time that second ps has been executed the new connmand has exited.
This is a classic race condition and on a heavily loaded autobuilder inserting
an arbitrary sleep isn't wise. In the scheme of things this test isn't very
useful, so delete it.
(From OE-Core rev: 80ef721140c79e29430d0a5692a5c176db0061e6)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Builds a simple Hello World module on target.
Added to the defaults for core-image-sato-sdk.
(From OE-Core rev: c61c3dee162aa1f5bf31b2a09d8b916dc1712056)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of considering that ping test passed after 1 reply,
wait for at least 5 consecutive replies in 60 seconds (which should
be enough time for connman to reconfigure the interface in systemd
images and help with the fake ssh/tests fails.)
(From OE-Core rev: 6fd19a9df0ad25b2822f12e2c3a97f1b71068d4e)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- move everything in the same test. setUp/tearDown aren't quite
the right thing here, everything it's part of the same test. (and
it get's confusing when ssh fails)
ldd:
- change test name and add output to error message
vnc:
- remove unnecessary check as there is no point in doing
both ps and netstat. Also improve error output a bit.
(From OE-Core rev: 2f91bb438cfcdd0a40daed5902b6e98fc0aee67f)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Increase the timeout for smart commands as under load for qemumips
it's still to small. Also give ping more time fixing a potential
timeout for sato systemd.
(From OE-Core rev: daa3ad5807f6fc0d15b9310937d07a16edac6d22)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Don't add inappropiate channels on the target.
This happens when building two different machines in the
same dir and then running the tests for each machine.
(From OE-Core rev: 9e4e475b598f6b9bd2b653c72a1c1d5bd5c0b8c9)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This uses scanelf from the pax-utils package and scans the binaries in PATH
for TEXTREL and RPATH information. For a sato image with pax-utils installed
it shows no output (which is good).
(From OE-Core rev: 629099ad66f5fa2814e5f7908b426149e8978e43)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use a file object to generate a our test file instead of calling `dd`;
removes dd's output from testimage.log, keeps unittest output clean.
Also remove unused imports.
(From OE-Core rev: 6ac48ffbab29a37b0eada533191878aeae3c91f0)
Signed-off-by: Mihai Lindner <mihaix.lindner@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Set longMessage to True for all tests derived from oeRuntimeTest, in
order to have somewhat info on assertions with cryptic or no messages.
(From OE-Core rev: e5196ebd6c652431a3b80cdc9f60b752401314f4)
Signed-off-by: Mihai Lindner <mihaix.lindner@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Don't start the http server on 0.0.0.0, listen on host ip (end of tap interface) only.
Also use the timeout option (default is 300s for ssh commands) for all the commands
run in this module (mostly because smart update timeouts on mips).
(From OE-Core rev: 8c272641ef3e8410f331ca4133d28dea8f36e4f4)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We run the ping test as soon as we reach the login prompt.
But sometimes (seen in sato systemd) we end up with link down/link up stuff like:
qemux86-64 login: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
The logic behind ping -w 30 -c 1 was to wait at most 30 seconds
for at least one reply, but there is a catch: reply doesn't seems
to be echo reply but any reply (non-reply means loss not network error)
ping's man page:
-w deadline
Specify a timeout, in seconds, before ping exits regardless of
how many packets have been sent or received. In this case ping
does not stop after count packet are sent, it waits either for
deadline expire or until count probes are answered or for some
error notification from network.
Just when the link up/link down happens ping returns:
From 192.168.7.1 icmp_seq=1 Destination Host Unreachable
--- 192.168.7.2 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms
and exits sooner than the 30 seconds timeout.
This patch should do what was originally intended (wait at most
30 seconds for at least one reply).
(From OE-Core rev: 56d144fd22d37189e49cdf3032afb00f0be469c6)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Recently syslog behaviour changed for systemd images (log
it's in a buffer not in /var/log/messages), account
for the new stuff.
(From OE-Core rev: 32576c4cc1621fa3013eac66c7caaa1e1fd14995)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
OE-core commit fcc59cbcdb1550489d372edf9f465efa7165245f /
poky commit 748ddc39e5 added a new test, but
in the wrong location.
I took the patch from Alex's branch but renamed it from meta/lib/oeqa/runtime/skeleton.py to meta/lib/oeqa/skeletoninit.py before sending. This was
unintentional, it should have been under meta/lib/oeqa/runtime.
(From OE-Core rev: f12c346ef48cb44be2e356e4cf4f28d015c3f507)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A layer can add tests in lib/oeqa/runtime (provided it extends BBPATH as
normal) and enable them with TEST_SUITES_append = " testname". Test
module names shouldn't collide though.
(From OE-Core rev: e1e347a2d509303e1c566450b0f2b485d3d6629f)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These checks are unnecessary.
setUpModule is run when a module is loaded and we
shouldn't run commands on the target here, (plus if
ssh doesn't work we error out in setup multiple times, instead
of skipping the real test, which might depend on test_ssh).
(From OE-Core rev: 188acd0a75e188fd7c0d2979acaf13fd18b12106)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sometimes smart throws:
Committing transaction...
Preparing... ######################################## [ 0%]
1:Removing psplash-default ######################################## [100%]
error: Couldn't fork %postun: Cannot allocate memory
and returns a 0 exit code (it thinks it succesfully removed the package,
when in reality it didn't), so we need to catch those specifically.
Also, sometimes output from download command is:
Saving cache...http://192.168.7.1:49456/rpm/x86_64_x32/psplash-default-0.1+git0+afd4e228c6-r15.x86_64_x32.rpm
and that tricks our smart download test, so use a regex there.
(From OE-Core rev: 2ac7783e04f5e8e6005f967e1a6dd65d2fc6a19a)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The check was obviously wrong and it surfaced
with the recent change in behaviour for skipping tests.
(From OE-Core rev: 4a14535cd493cb2bdd46b2a5f2a1cd2b38161f0a)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Downloads iptables/cvs/sudoku-savant sources and builds them on target.
(From OE-Core rev: df4568205c3a7e0b20c6299e29f96bd30560146b)
Signed-off-by: Mihai Prica <mihai.prica@intel.com>
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add class to be inherited by smart tests, along with more basic tests and tests
using a rpm repository.
(From OE-Core rev: f6186b4204dcc421b4e616774315c8a2a77fb5c5)
Signed-off-by: Mihai Lindner <mihaix.lindner@linux.intel.com>
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Copies to target rpm-doc file from deploy_dir
and tries to install and then remove that package.
rpm-doc was chosen because it's small, it only adds
a few files to target, and it's almost always found in
deploy_dir for images with package-management/rpm.
(From OE-Core rev: a2d2f2b7b111863d3c50dedded37aab813d9634f)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Copies a 5MB to target using scp, more of an
network test than a scp one.
(From OE-Core rev: 2ec4a0686b9a91e56dfba3fa2e574c0c531508ff)
Signed-off-by: Alexandru Palalau <alexandrux.palalau@intel.com>
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
New test which verifies some usual commands functionality with PAM support
(From OE-Core rev: 06c7914e3354ff4e430a6b664f40e5a71e212761)
Signed-off-by: Alexandru Palalau <alexandrux.palalau@intel.com>
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
New logrotate test which verifies the log directory change in logrotate.conf.
Needs an image with logrotate installed.
(From OE-Core rev: 186d79a603b5cbf5a93e6f5dbba5f62ed8d4d8d8)
Signed-off-by: Alexandru Palalau <alexandrux.palalau@intel.com>
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This test checks that at least one path in RTLDLIST exists.
(From OE-Core rev: d53ffbf21a6b17f0f0537929a95a502958d82a09)
Signed-off-by: Cornel Stoicescu <corneliux.stoicescu@intel.com>
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This test runs a perl script on the target and checks the output.
(From OE-Core rev: 95abdbf2d715c9e7e3368f5193167feb5efc6261)
Signed-off-by: Cornel Stoicescu <corneliux.stoicescu@intel.com>
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Checks that an x86-64-x32 image has the right binaries.
(From OE-Core rev: da42e38b2bc4a606b717ac302cfc77aefb1a5795)
Signed-off-by: Alexandru Georgescu <alexandru.c.georgescu@intel.com>
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
New systemd runtime tests for enable/disable service, start/stop service and list services.
(From OE-Core rev: 6386dc718f85210c9b6b9f69878ec9a7847b78de)
Signed-off-by: Alexandru Palalau <alexandrux.palalau@intel.com>
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add separate tests for restarting syslog and using logger, and
skip the configuration test for systemd images which always fail
because syslog's systemd service doesn't read a config by default
(see YB#4860).
(From OE-Core rev: c75f3e2385dde44ee96e33f4e5d064894dfb7d52)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use our determined ps command, otherwise test fails for images with procps.
(From OE-Core rev: 18e4d014bea3a5748f42e6dac648fa9b02230344)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds missing skip for smart test and fix the check (which I somehow broke
a while ago).
(From OE-Core rev: cf1790d992f067be8d5f9894458f55f6f1bdc61f)
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add tests for free space and syslog.
Changed in v2:
- limit df's output to /
- syslog: fix restart in case of systemd
(From OE-Core rev: 1b39d57e7b5c9b69d565cf4d188ebc2f14e66ae6)
Signed-off-by: Alexandru Palalau <alexandru.palalau@intel.com>
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Stefan Stanacar
Richard Purdie
Ross Burton
Ting Wang
Saul Wold
Mihai Lindner
Mihai Prica
Alexandru Palalau
Cornel Stoicescu
Alexandru Georgescu
Alexandru Palalau