ref-manual: Added section on test and QA infrastructure

Fixes [YOCTO #10887]

There needed to be an overview of the test and QA infrastructure
used by the YP development team for releases and that is visible
to the developer.  I added a new section for this information into
the existing section on YP release processes.

(From yocto-docs rev: 17b3d6898b37bdbfbd23a333a4d659a9746696d0)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is contained in:
Scott Rifenbark 2017-03-30 07:51:19 -07:00 committed by Richard Purdie
parent c53ba427ef
commit 43fefd1211
1 changed files with 131 additions and 0 deletions

View File

@ -117,6 +117,137 @@
</para>
</section>
<section id='testing-and-quality-assurance'>
<title>Testing and Quality Assurance</title>
<para>
Part of the Yocto Project development and release process is quality
assurance through the execution of test strategies.
Test strategies provide the Yocto Project team a way to ensure a
release is validated.
Additionally, because the test strategies are visible to you as a
developer, you can validate your projects.
This section overviews the available test infrastructure used in the
Yocto Project.
For information on how to run available tests on your projects, see the
"<ulink url='&YOCTO_DOCS_DEV_URL;#performing-automated-runtime-testing'>Performing Automated Runtime Testing</ulink>"
section in the Yocto Project Development Manual.
</para>
<para>
The QA/testing infrastructure is woven into the project to the point
where core developers take some of it for granted.
The infrastructure consists of the following pieces:
<itemizedlist>
<listitem><para>
<filename>bitbake-selftest</filename>:
A standalone command that runs unit tests on key pieces of
BitBake and its fetchers.
</para></listitem>
<listitem><para>
<link linkend='ref-classes-sanity'><filename>sanity.bbclass</filename></link>:
This automatically included class checks the build environment
for missing tools (e.g. <filename>gcc</filename>) or common
misconfigurations such as
<link linkend='var-MACHINE'><filename>MACHINE</filename></link>
set incorrectly.
</para></listitem>
<listitem><para>
<link linkend='ref-classes-insane'><filename>insane.bbclass</filename></link>:
This class checks the generated output from builds for sanity.
For example, if building for an ARM target, did the build
produce ARM binaries.
If, for example, the build produced PPC binaries then there
is a problem.
</para></listitem>
<listitem><para>
<link linkend='ref-classes-testimage*'><filename>testimage.bbclass</filename></link>:
This class performs runtime testing of images after they are
built.
The tests are usually used with
<ulink url='&YOCTO_DOCS_DEV_URL;#dev-manual-qemu'>QEMU</ulink>
to boot the images and check the combined runtime result
boot operation and functions.
However, the test can also use the IP address of a machine to
test.
</para></listitem>
<listitem><para>
<ulink url='&YOCTO_DOCS_DEV_URL;#testing-packages-with-ptest'><filename>ptest</filename></ulink>:
Runs tests against packages produced during the build for a
given piece of software.
The test allows the packages to be be run within a target
image.
</para></listitem>
<listitem><para>
<filename>oe-selftest</filename>:
Tests combination BitBake invocations.
These tests operate outside the OpenEmbedded build system
itself.
The <filename>oe-selftest</filename> can run all tests by
default or can run selected tests or test suites.
<note>
Running <filename>oe-selftest</filename> requires
host packages beyond the "Essential" grouping.
See the
"<link linkend='required-packages-for-the-host-development-system'>Required Packages for the Host Development System</link>"
section for more information.
</note>
</para></listitem>
</itemizedlist>
</para>
<para>
Originally, much of this testing was done manually.
However, significant effort has been made to automate the tests so
that more people can use them and the Yocto Project development team
can run them faster and more efficiently.
</para>
<para>
The Yocto Project's main Autobuilder
(<filename>autobuilder.yoctoproject.org</filename>) publicly tests
each Yocto Project release's code in the OE-Core, Poky, and BitBake
repositories.
The testing occurs for both the current state of the
"master" branch and also for submitted patches.
Testing for submitted patches usually occurs in the
"ross/mut" branch in the <filename>poky-contrib</filename> repository
(i.e. the master-under-test branch) or in the "master-next" branch
in the <filename>poky</filename> repository.
<note>
You can find all these branches in the Yocto Project
<ulink url='&YOCTO_DOCS_DEV_URL;#source-repositories'>Source Repositories</ulink>.
</note>
Testing within these public branches ensures in a publicly visible way
that all of the main supposed architectures and recipes in OE-Core
successfully build and behave properly.
</para>
<para>
Various features such as <filename>multilib</filename>, sub
architectures (e.g. <filename>x32</filename>,
<filename>poky-tiny</filename>, <filename>musl</filename>,
<filename>no-x11</filename> and and so forth),
<filename>bitbake-selftest</filename>, and
<filename>oe-selftest</filename> are tested as part of
the QA process of a release.
Complete testing and validation for a release takes the Autobuilder
workers several hours.
<note>
The Autobuilder workers are non-homogeneous, which means regular
testing across a variety of Linux distributions occurs.
The Autobuilder is limited to only testing QEMU-based setups and
not real hardware.
</note>
</para>
<para>
Finally, in addition to the Autobuilder's tests, the Yocto Project
QA team also performs testing on a variety of platforms, which includes
actual hardware, to ensure expected results.
</para>
</section>
</chapter>
<!--
vim: expandtab tw=80 ts=4