sysmo-bts
/
generic-poky
9
0
Fork 0
Code Pull Requests Releases Activity

ref-manual, dev-manual: Applied review comments for test section 

The section and variables for performing automated runtime tests
on images was reviewed by Paul Eggleton.  The suggested changes
were made.

(From yocto-docs rev: c2f84ea3c162892e4da3df30fb833f88bab3d3cc)

Signed-off-by: Scott Rifenbark <scott.m.rifenbark@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is contained in:
Scott Rifenbark 2013-09-23 09:19:53 -07:00 committed by Richard Purdie
parent 56c4de93b9
commit e44538f4e4
3 changed files with 109 additions and 81 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

113
documentation/dev-manual/dev-manual-common-tasks.xml
View File

@ -4252,24 +4252,22 @@
</section> </section>
</section> </section>
<section id="running-and-writing-tests-for-a-qemu-image"> <section id="performing-automated-runtime-testing">
<title>Running and Writing Tests for a QEMU Image</title> <title>Performing Automated Runtime Testing</title>
<para> <para>
The OpenEmbedded build system makes available a series of automated The OpenEmbedded build system makes available a series of automated
tests for QEMU images. tests for images.
These tests are commands that run on the target system over <note>
<filename>ssh</filename> and are written in Python, thus making The current release of Yocto Project supports these tests
use of the <filename>unittest</filename> module. for QEMU images only.
</para> </note>
These tests are written in Python making use of the
<para> <filename>unittest</filename> module, and the majority of them
run commands on the target system over
<filename>ssh</filename>.
This section describes how you set up the environment to use these This section describes how you set up the environment to use these
tests, run available tests, and write and add your own tests. tests, run available tests, and write and add your own tests.
You can the same information in a different form and with example
log files of test runs on the
<ulink url='https://wiki.yoctoproject.org/wiki/Image_tests'>Image Tests</ulink>
Wiki page.
</para> </para>
<section id="qemu-image-enabling-tests"> <section id="qemu-image-enabling-tests">
@ -4278,9 +4276,8 @@
<para> <para>
In order to run tests, you need to do the following: In order to run tests, you need to do the following:
<itemizedlist> <itemizedlist>
<listitem><para><emphasis>Ensure you run the test <listitem><para><emphasis>Set up to avoid interaction
automatically with no interaction and under with <filename>sudo</filename> for networking:</emphasis>
<filename>sudo</filename>:</emphasis>
To accomplish this, you must do one of the To accomplish this, you must do one of the
following: following:
<itemizedlist> <itemizedlist>
@ -4300,12 +4297,13 @@
<listitem><para>Manually configure a tap interface <listitem><para>Manually configure a tap interface
for your system.</para></listitem> for your system.</para></listitem>
<listitem><para>Run as root the script in <listitem><para>Run as root the script in
<filename>scripts/runqemu-gen-tapdev</filename>, <filename>scripts/runqemu-gen-tapdevs</filename>,
which should generate a list of tap devices. which should generate a list of tap devices.
List generation is usually done in environments This is the option typically chosen for
similar to AutoBuilder.</para></listitem> Autobuilder-type environments.
</para></listitem>
</itemizedlist></para></listitem> </itemizedlist></para></listitem>
<listitem><para><emphasis>Set up the <listitem><para><emphasis>Set the
<filename>DISPLAY</filename> variable:</emphasis> <filename>DISPLAY</filename> variable:</emphasis>
You need to set this variable so that you have an X You need to set this variable so that you have an X
server available (e.g. start server available (e.g. start
@ -4321,11 +4319,10 @@
<filename>${DEPLOY_DIR}/rpm</filename> so it can run <filename>${DEPLOY_DIR}/rpm</filename> so it can run
smart channel commands. That means your host's firewall smart channel commands. That means your host's firewall
must accept incoming connections from 192.168.7.0/24, must accept incoming connections from 192.168.7.0/24,
which is the default class used for which is the default IP range used for tap devices
<filename>tap0</filename> devices by by <filename>runqemu</filename>.</para></listitem>
<filename>runqemu</filename>.</para></listitem> <listitem><para><emphasis>Have your image built:</emphasis>
<listitem><para><emphasis>Have your QEMU image built:</emphasis> The image on which you want to run tests needs to
The QEMU image on which you want to run tests needs to
be built. be built.
For example, the following command would build the For example, the following command would build the
<filename>core-image-sato</filename> image when <filename>core-image-sato</filename> image when
@ -4333,18 +4330,34 @@
<literallayout class='monospaced'> <literallayout class='monospaced'>
bitbake core-image-sato bitbake core-image-sato
</literallayout></para></listitem> </literallayout></para></listitem>
<listitem><para><emphasis>Globally inherit <listitem><para><emphasis>Run the tests:</emphasis>
<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-testimage'><filename>testimage.class</filename>:</ulink></emphasis> You can start the tests automatically or manually.
Edit your <filename>local.conf</filename> file as To have the tests start automatically after the
OpenEmbedded build system successfully creates a QEMU
image, set the
<ulink url='&YOCTO_DOCS_REF_URL;#var-TEST_IMAGE'><filename>TEST_IMAGE</filename></ulink>
variable to "1" in your
<filename>local.conf</filename> file in the
<ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>.
To manually run the tests, globally inherit
<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-testimage'><filename>testimage.class</filename>:</ulink>
by editing your <filename>local.conf</filename> file as
follows: follows:
<literallayout class='monospaced'> <literallayout class='monospaced'>
INHERIT += "testimage" INHERIT += "testimage"
</literallayout>
After editing your
<filename>local.conf</filename> file, use BitBake to
run the tests:
<literallayout class='monospaced'>
bitbake -c testimage &lt;qemu-image&gt;
</literallayout></para></listitem> </literallayout></para></listitem>
</itemizedlist> </itemizedlist>
</para> </para>
<note> <note>
If you built your QEMU image using <filename>rm_work</filename>, Regardless of how you initiate the tests, if you built your
image using <filename>rm_work</filename>,
most of the tests will fail with errors because they rely on most of the tests will fail with errors because they rely on
<filename>${WORKDIR}/installed_pkgs.txt</filename>. <filename>${WORKDIR}/installed_pkgs.txt</filename>.
</note> </note>
@ -4355,14 +4368,16 @@
<para> <para>
After setting up to run the tests, use BitBake to start After setting up to run the tests, use BitBake to start
the standard suite of tests: the standard suite of tests manually or, if you are running
them automatically, build your image:
<literallayout class='monospaced'> <literallayout class='monospaced'>
bitbake core-image-sato -c testimage bitbake core-image-sato -c testimage
bitbake core-image-sato
</literallayout> </literallayout>
</para> </para>
<para> <para>
Once you start the tests, the following happens: Once the tests start, the following happens:
<itemizedlist> <itemizedlist>
<listitem><para>A copy of the root filesystem is written <listitem><para>A copy of the root filesystem is written
to <filename>${WORKDIR}/testimage</filename>. to <filename>${WORKDIR}/testimage</filename>.
@ -4400,10 +4415,10 @@
All test files reside in All test files reside in
<filename>meta/lib/oeqa/runtime</filename> in the <filename>meta/lib/oeqa/runtime</filename> in the
<link linkend='source-directory'>Source Directory</link>. <link linkend='source-directory'>Source Directory</link>.
Tests (also referred to as modules) have a one-to-one A test name (sometimes referred to as a module) maps to a
relationship with the filenames in <filename>runtime</filename>. method name within a class within a module.
Modules can have multiple classes and test methods and are Tests can be used within multiple classes and the tests
usually grouped together by the area tested (e.g tests for are usually grouped together by the area tested (e.g tests for
<filename>systemd</filename> reside in <filename>systemd</filename> reside in
<filename>meta/lib/oeqa/runtime/systemd.py</filename>). <filename>meta/lib/oeqa/runtime/systemd.py</filename>).
</para> </para>
@ -4412,12 +4427,13 @@
You can add tests to any layer provided you place them in the You can add tests to any layer provided you place them in the
proper area and you extend proper area and you extend
<ulink url='&YOCTO_DOCS_REF_URL;#var-BBPATH'><filename>BBPATH</filename></ulink> <ulink url='&YOCTO_DOCS_REF_URL;#var-BBPATH'><filename>BBPATH</filename></ulink>
in the <filename>local.conf</filename> file. in the <filename>local.conf</filename> file as normal.
Be sure that tests reside in Be sure that tests reside in
<filename>&lt;layer&gt;/lib/oeqa/runtime</filename>. <filename>&lt;layer&gt;/lib/oeqa/runtime</filename>.
<note> <note>
Be sure that module names do not collide with module names Be sure that module names do not collide with module names
provided by the Yocto Project. used in the default set of test modules in
<filename>meta/lib/oeqa/runtime</filename>.
</note> </note>
</para> </para>
@ -4458,12 +4474,12 @@
<listitem><para>Run the default set of tests simply by <listitem><para>Run the default set of tests simply by
running the following: running the following:
<literallayout class='monospaced'> <literallayout class='monospaced'>
bitbake &lt;qemu_image&gt; -c testimage bitbake &lt;image&gt; -c testimage
</literallayout></para></listitem> </literallayout></para></listitem>
<listitem><para>The default tests for the image are defined <listitem><para>The default tests for the image are defined
as: as:
<literallayout class='monospaced'> <literallayout class='monospaced'>
DEFAULT_TEST_SUITES_pn-&lt;qemu_image&gt; = "ping ssh df connman syslog xorg scp vnc date rpm smart dmesg" DEFAULT_TEST_SUITES_pn-&lt;image&gt; = "ping ssh df connman syslog xorg scp vnc date rpm smart dmesg"
</literallayout></para></listitem> </literallayout></para></listitem>
<listitem><para>Add your own test to the list of the <listitem><para>Add your own test to the list of the
by using the following: by using the following:
@ -4487,15 +4503,16 @@
<para> <para>
As mentioned previously, all new test files need to be in the As mentioned previously, all new test files need to be in the
proper place for the build system to find them. proper place for the build system to find them.
New tests can go in the New tests for additional functionality outside of the core
<link linkend='source-directory'>Source Directory</link> at should be added to the layer that adds the functionality, in
<filename>meta/lib/oeqa/runtime</filename>. <filename>&lt;layer&gt;/lib/oeqa/runtime</filename> (as
Alternatively, a layer can add its own tests in long as
<filename>&lt;layer&gt;/lib/oeqa/runtime</filename> as long <ulink url='&YOCTO_DOCS_REF_URL;#var-BBPATH'><filename>BBPATH</filename></ulink>
as you extend <filename>BBPATH</filename>. is extended in the layer's
Just remember that filenames need to map directory to test <filename>layer.conf</filename> file as normal.
(module) names and you not use module names that collide with Just remember that filenames need to map directly to test
existing core tests. (module) names and that you do not use module names that
collide with existing core tests.
</para> </para>
<para> <para>

16
documentation/ref-manual/ref-classes.xml
View File

@ -940,8 +940,12 @@
<para> <para>
You can use this class to enable running a series of automated tests You can use this class to enable running a series of automated tests
for QEMU images. for images.
The class handles loading the tests and starting the QEMU image. The class handles loading the tests and starting the image.
<note>
The current release of Yocto Project supports these tests
for QEMU images only.
</note>
</para> </para>
<para> <para>
@ -950,15 +954,13 @@
The tests are commands that run on the target system over The tests are commands that run on the target system over
<filename>ssh</filename>. <filename>ssh</filename>.
they are written in Python and make use of the they are written in Python and make use of the
<filename>unittest modulebuild</filename> software. <filename>unittest</filename> module.
</para> </para>
<para> <para>
For information on how to enable, run, and create new tests, see the For information on how to enable, run, and create new tests, see the
"<ulink url='running-and-writing-tests-for-a-qemu-image'>Running and Writing Tests for a QEMU Image</ulink>" "<ulink url='&YOCTO_DOCS_DEV_URL;#performing-automated-runtime-testing'>Performing Automated Runtime Testing</ulink>"
section and the section.
<ulink url='https://wiki.yoctoproject.org/wiki/Image_tests'>Image Tests</ulink>
Wiki page.
</para> </para>
</section> </section>

61
documentation/ref-manual/ref-variables.xml
View File

@ -5220,21 +5220,27 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
<glossentry id='var-TEST_IMAGE'><glossterm>TEST_IMAGE</glossterm> <glossentry id='var-TEST_IMAGE'><glossterm>TEST_IMAGE</glossterm>
<glossdef> <glossdef>
<para> <para>
Automatically runs the series of automated tests for QEMU Automatically runs the series of automated tests for
images when a QEMU image is successfully built. images when an image is successfully built.
These tests are commands that run on the target system over <note>
<filename>ssh</filename> and are written in Python. The current release of Yocto Project supports these tests
for QEMU images only.
</note>
These tests are written in Python making use of the
<filename>unittest</filename> module, and the majority of
them run commands on the target system over
<filename>ssh</filename>.
You can set this variable to "1" in your You can set this variable to "1" in your
<filename>local.conf</filename> file in the <filename>local.conf</filename> file in the
<ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink> <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>
to have the OpenEmbedded build system automatically run to have the OpenEmbedded build system automatically run
these tests after a QEMU image successfully builds: these tests after an image successfully builds:
<literallayout class='monospaced'> <literallayout class='monospaced'>
TEST_IMAGE = "1" TEST_IMAGE = "1"
</literallayout> </literallayout>
For more information on enabling, running, and writing For more information on enabling, running, and writing
these tests, see the these tests, see the
"<ulink url='&YOCTO_DOCS_DEV_URL;#running-and-writing-tests-for-a-qemu-image'>Running and Writing Tests for a QEMU Image</ulink>" "<ulink url='&YOCTO_DOCS_DEV_URL;#performing-automated-runtime-testing'>Performing Automated Runtime Testing</ulink>"
section in the Yocto Project Development Manual and the section in the Yocto Project Development Manual and the
"<link linkend='ref-classes-testimage'><filename>testimage.bbclass</filename></link>" "<link linkend='ref-classes-testimage'><filename>testimage.bbclass</filename></link>"
section. section.
@ -5245,24 +5251,18 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
<glossentry id='var-TEST_QEMUBOOT_TIMEOUT'><glossterm>TEST_QEMUBOOT_TIMEOUT</glossterm> <glossentry id='var-TEST_QEMUBOOT_TIMEOUT'><glossterm>TEST_QEMUBOOT_TIMEOUT</glossterm>
<glossdef> <glossdef>
<para> <para>
The time in seconds allowed for a QEMU image to boot before The time in seconds allowed for an image to boot before
tests begin to run against the image using the following automated runtime tests begin to run against an
command: image.
<literallayout class='monospaced'>
bitbake &lt;qemu_image&gt; -c testimage
</literallayout>
</para>
<para>
The default timeout period to allow the boot process to The default timeout period to allow the boot process to
reach the login prompt is 500 seconds. reach the login prompt is 500 seconds.
You can edit this default value in the You can specify a different value in the
<filename>local.conf</filename> file. <filename>local.conf</filename> file.
</para> </para>
<para> <para>
For more information on testing QEMU images, see the For more information on testing images, see the
"<ulink url='running-and-writing-tests-for-a-qemu-image'>Running and Writing Tests for a QEMU Image</ulink>" "<ulink url='&YOCTO_DOCS_DEV_URL;#performing-automated-runtime-testing'>Performing Automated Runtime Testing</ulink>"
section in the Yocto Project Development Manual. section in the Yocto Project Development Manual.
</para> </para>
</glossdef> </glossdef>
@ -5272,15 +5272,16 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
<glossdef> <glossdef>
<para> <para>
An ordered list of tests (modules) to run against An ordered list of tests (modules) to run against
a QEMU image using the following command: an image when performing automated runtime testing.
<literallayout class='monospaced'>
bitbake &lt;qemu_image&gt; -c testimage
</literallayout>
</para> </para>
<para> <para>
The Yocto Project provides a core set of tests that can The OpenEmbedded build system provides a core set of tests
be used against QEMU images. that can be used against images.
<note>
The current release of Yocto Project supports these
tests for QEMU images only.
</note>
Tests include <filename>ping</filename>, Tests include <filename>ping</filename>,
<filename>ssh</filename>, <filename>df</filename> among <filename>ssh</filename>, <filename>df</filename> among
others. others.
@ -5289,6 +5290,14 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
<literallayout class='monospaced'> <literallayout class='monospaced'>
TEST_SUITES_append = " mytest" TEST_SUITES_append = " mytest"
</literallayout> </literallayout>
Alternatively, you can provide the "auto" option to
have all applicable tests run against the image.
<literallayout class='monospaced'>
TEST_SUITES_append = " auto"
</literallayout>
Using this option causes the build system to automatically
run tests that are applicable to the image.
Tests that are not applicable are skipped.
</para> </para>
<para> <para>
@ -5307,8 +5316,8 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
</para> </para>
<para> <para>
For more information on testing QEMU images, see the For more information on testing images, see the
"<ulink url='running-and-writing-tests-for-a-qemu-image'>Running and Writing Tests for a QEMU Image</ulink>" "<ulink url='&YOCTO_DOCS_DEV_URL;#performing-automated-runtime-testing'>Performing Automated Runtime Testing</ulink>"
section in the Yocto Project Development Manual. section in the Yocto Project Development Manual.
</para> </para>
</glossdef> </glossdef>