diff --git a/documentation/ref-manual/ref-release-process.xml b/documentation/ref-manual/ref-release-process.xml index 1b337f8bd7..fe3ba0933e 100644 --- a/documentation/ref-manual/ref-release-process.xml +++ b/documentation/ref-manual/ref-release-process.xml @@ -117,6 +117,137 @@ +
+ Testing and Quality Assurance + + + Part of the Yocto Project development and release process is quality + assurance through the execution of test strategies. + Test strategies provide the Yocto Project team a way to ensure a + release is validated. + Additionally, because the test strategies are visible to you as a + developer, you can validate your projects. + This section overviews the available test infrastructure used in the + Yocto Project. + For information on how to run available tests on your projects, see the + "Performing Automated Runtime Testing" + section in the Yocto Project Development Manual. + + + + The QA/testing infrastructure is woven into the project to the point + where core developers take some of it for granted. + The infrastructure consists of the following pieces: + + + bitbake-selftest: + A standalone command that runs unit tests on key pieces of + BitBake and its fetchers. + + + sanity.bbclass: + This automatically included class checks the build environment + for missing tools (e.g. gcc) or common + misconfigurations such as + MACHINE + set incorrectly. + + + insane.bbclass: + This class checks the generated output from builds for sanity. + For example, if building for an ARM target, did the build + produce ARM binaries. + If, for example, the build produced PPC binaries then there + is a problem. + + + testimage.bbclass: + This class performs runtime testing of images after they are + built. + The tests are usually used with + QEMU + to boot the images and check the combined runtime result + boot operation and functions. + However, the test can also use the IP address of a machine to + test. + + + ptest: + Runs tests against packages produced during the build for a + given piece of software. + The test allows the packages to be be run within a target + image. + + + oe-selftest: + Tests combination BitBake invocations. + These tests operate outside the OpenEmbedded build system + itself. + The oe-selftest can run all tests by + default or can run selected tests or test suites. + + Running oe-selftest requires + host packages beyond the "Essential" grouping. + See the + "Required Packages for the Host Development System" + section for more information. + + + + + + + Originally, much of this testing was done manually. + However, significant effort has been made to automate the tests so + that more people can use them and the Yocto Project development team + can run them faster and more efficiently. + + + + The Yocto Project's main Autobuilder + (autobuilder.yoctoproject.org) publicly tests + each Yocto Project release's code in the OE-Core, Poky, and BitBake + repositories. + The testing occurs for both the current state of the + "master" branch and also for submitted patches. + Testing for submitted patches usually occurs in the + "ross/mut" branch in the poky-contrib repository + (i.e. the master-under-test branch) or in the "master-next" branch + in the poky repository. + + You can find all these branches in the Yocto Project + Source Repositories. + + Testing within these public branches ensures in a publicly visible way + that all of the main supposed architectures and recipes in OE-Core + successfully build and behave properly. + + + + Various features such as multilib, sub + architectures (e.g. x32, + poky-tiny, musl, + no-x11 and and so forth), + bitbake-selftest, and + oe-selftest are tested as part of + the QA process of a release. + Complete testing and validation for a release takes the Autobuilder + workers several hours. + + The Autobuilder workers are non-homogeneous, which means regular + testing across a variety of Linux distributions occurs. + The Autobuilder is limited to only testing QEMU-based setups and + not real hardware. + + + + + Finally, in addition to the Autobuilder's tests, the Yocto Project + QA team also performs testing on a variety of platforms, which includes + actual hardware, to ensure expected results. + +
+