ref-manual: Added section on test and QA infrastructure
Fixes [YOCTO #10887] There needed to be an overview of the test and QA infrastructure used by the YP development team for releases and that is visible to the developer. I added a new section for this information into the existing section on YP release processes. (From yocto-docs rev: 17b3d6898b37bdbfbd23a333a4d659a9746696d0) Signed-off-by: Scott Rifenbark <srifenbark@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is contained in:
parent
c53ba427ef
commit
43fefd1211
|
@ -117,6 +117,137 @@
|
||||||
</para>
|
</para>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
|
<section id='testing-and-quality-assurance'>
|
||||||
|
<title>Testing and Quality Assurance</title>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
Part of the Yocto Project development and release process is quality
|
||||||
|
assurance through the execution of test strategies.
|
||||||
|
Test strategies provide the Yocto Project team a way to ensure a
|
||||||
|
release is validated.
|
||||||
|
Additionally, because the test strategies are visible to you as a
|
||||||
|
developer, you can validate your projects.
|
||||||
|
This section overviews the available test infrastructure used in the
|
||||||
|
Yocto Project.
|
||||||
|
For information on how to run available tests on your projects, see the
|
||||||
|
"<ulink url='&YOCTO_DOCS_DEV_URL;#performing-automated-runtime-testing'>Performing Automated Runtime Testing</ulink>"
|
||||||
|
section in the Yocto Project Development Manual.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
The QA/testing infrastructure is woven into the project to the point
|
||||||
|
where core developers take some of it for granted.
|
||||||
|
The infrastructure consists of the following pieces:
|
||||||
|
<itemizedlist>
|
||||||
|
<listitem><para>
|
||||||
|
<filename>bitbake-selftest</filename>:
|
||||||
|
A standalone command that runs unit tests on key pieces of
|
||||||
|
BitBake and its fetchers.
|
||||||
|
</para></listitem>
|
||||||
|
<listitem><para>
|
||||||
|
<link linkend='ref-classes-sanity'><filename>sanity.bbclass</filename></link>:
|
||||||
|
This automatically included class checks the build environment
|
||||||
|
for missing tools (e.g. <filename>gcc</filename>) or common
|
||||||
|
misconfigurations such as
|
||||||
|
<link linkend='var-MACHINE'><filename>MACHINE</filename></link>
|
||||||
|
set incorrectly.
|
||||||
|
</para></listitem>
|
||||||
|
<listitem><para>
|
||||||
|
<link linkend='ref-classes-insane'><filename>insane.bbclass</filename></link>:
|
||||||
|
This class checks the generated output from builds for sanity.
|
||||||
|
For example, if building for an ARM target, did the build
|
||||||
|
produce ARM binaries.
|
||||||
|
If, for example, the build produced PPC binaries then there
|
||||||
|
is a problem.
|
||||||
|
</para></listitem>
|
||||||
|
<listitem><para>
|
||||||
|
<link linkend='ref-classes-testimage*'><filename>testimage.bbclass</filename></link>:
|
||||||
|
This class performs runtime testing of images after they are
|
||||||
|
built.
|
||||||
|
The tests are usually used with
|
||||||
|
<ulink url='&YOCTO_DOCS_DEV_URL;#dev-manual-qemu'>QEMU</ulink>
|
||||||
|
to boot the images and check the combined runtime result
|
||||||
|
boot operation and functions.
|
||||||
|
However, the test can also use the IP address of a machine to
|
||||||
|
test.
|
||||||
|
</para></listitem>
|
||||||
|
<listitem><para>
|
||||||
|
<ulink url='&YOCTO_DOCS_DEV_URL;#testing-packages-with-ptest'><filename>ptest</filename></ulink>:
|
||||||
|
Runs tests against packages produced during the build for a
|
||||||
|
given piece of software.
|
||||||
|
The test allows the packages to be be run within a target
|
||||||
|
image.
|
||||||
|
</para></listitem>
|
||||||
|
<listitem><para>
|
||||||
|
<filename>oe-selftest</filename>:
|
||||||
|
Tests combination BitBake invocations.
|
||||||
|
These tests operate outside the OpenEmbedded build system
|
||||||
|
itself.
|
||||||
|
The <filename>oe-selftest</filename> can run all tests by
|
||||||
|
default or can run selected tests or test suites.
|
||||||
|
<note>
|
||||||
|
Running <filename>oe-selftest</filename> requires
|
||||||
|
host packages beyond the "Essential" grouping.
|
||||||
|
See the
|
||||||
|
"<link linkend='required-packages-for-the-host-development-system'>Required Packages for the Host Development System</link>"
|
||||||
|
section for more information.
|
||||||
|
</note>
|
||||||
|
</para></listitem>
|
||||||
|
</itemizedlist>
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
Originally, much of this testing was done manually.
|
||||||
|
However, significant effort has been made to automate the tests so
|
||||||
|
that more people can use them and the Yocto Project development team
|
||||||
|
can run them faster and more efficiently.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
The Yocto Project's main Autobuilder
|
||||||
|
(<filename>autobuilder.yoctoproject.org</filename>) publicly tests
|
||||||
|
each Yocto Project release's code in the OE-Core, Poky, and BitBake
|
||||||
|
repositories.
|
||||||
|
The testing occurs for both the current state of the
|
||||||
|
"master" branch and also for submitted patches.
|
||||||
|
Testing for submitted patches usually occurs in the
|
||||||
|
"ross/mut" branch in the <filename>poky-contrib</filename> repository
|
||||||
|
(i.e. the master-under-test branch) or in the "master-next" branch
|
||||||
|
in the <filename>poky</filename> repository.
|
||||||
|
<note>
|
||||||
|
You can find all these branches in the Yocto Project
|
||||||
|
<ulink url='&YOCTO_DOCS_DEV_URL;#source-repositories'>Source Repositories</ulink>.
|
||||||
|
</note>
|
||||||
|
Testing within these public branches ensures in a publicly visible way
|
||||||
|
that all of the main supposed architectures and recipes in OE-Core
|
||||||
|
successfully build and behave properly.
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
Various features such as <filename>multilib</filename>, sub
|
||||||
|
architectures (e.g. <filename>x32</filename>,
|
||||||
|
<filename>poky-tiny</filename>, <filename>musl</filename>,
|
||||||
|
<filename>no-x11</filename> and and so forth),
|
||||||
|
<filename>bitbake-selftest</filename>, and
|
||||||
|
<filename>oe-selftest</filename> are tested as part of
|
||||||
|
the QA process of a release.
|
||||||
|
Complete testing and validation for a release takes the Autobuilder
|
||||||
|
workers several hours.
|
||||||
|
<note>
|
||||||
|
The Autobuilder workers are non-homogeneous, which means regular
|
||||||
|
testing across a variety of Linux distributions occurs.
|
||||||
|
The Autobuilder is limited to only testing QEMU-based setups and
|
||||||
|
not real hardware.
|
||||||
|
</note>
|
||||||
|
</para>
|
||||||
|
|
||||||
|
<para>
|
||||||
|
Finally, in addition to the Autobuilder's tests, the Yocto Project
|
||||||
|
QA team also performs testing on a variety of platforms, which includes
|
||||||
|
actual hardware, to ensure expected results.
|
||||||
|
</para>
|
||||||
|
</section>
|
||||||
|
|
||||||
</chapter>
|
</chapter>
|
||||||
<!--
|
<!--
|
||||||
vim: expandtab tw=80 ts=4
|
vim: expandtab tw=80 ts=4
|
||||||
|
|
Loading…
Reference in New Issue