Commit Graph

1034 Commits

Author SHA1 Message Date
Aníbal Limón 72e9ae377d oeqa/utils: Move targetbuild to buildproject module
The new buildproject module will contain only BuildProject class
a helper class for build source code.

The remaining classes TargetBuildProject and SDKBuildProject was
move to runtime and sdk respectively.

[YOCTO #10599]

(From OE-Core rev: 525fd2a5cda00890e921b63f7f608a10bc024d73)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:19 +00:00
Aníbal Limón 8a37763ae9 oeqa/sdk: Add case and context modules for the SDK component
Adds case and context modules for SDK based on oetest.py old code.

Enables SDK Test component usage with oe-test, the SDK Test component
adds command line options for specify sdk installed dir, sdk environment
and target/hosts maniftest.

[YOCTO #10599]

(From OE-Core rev: 19e875dd81c42841666e6db5f6b665b4e1cddfe6)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:19 +00:00
Aníbal Limón 9a23e0f703 oeqa/{runtime,sdk}/files: Move testsdkmakefile from runtime to sdk module
It doesn't make sense to have files related to sdk module into runtime
module.

[YOCTO #10599]

(From OE-Core rev: 9a3be58b17afbe4ef00030b0e6ad8b20b03adc49)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:19 +00:00
Aníbal Limón 2726c26efe oeqa/sdk: Move test cases inside cases directory
For match with the new structure of the OEQA framework.

In the new framework Test component base directory in this case
sdk module will contain case and context implementations.

[YOCTO #10599]

(From OE-Core rev: 57af8ee4021c302bd351adf03e6d85274ad7efd5)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:19 +00:00
Aníbal Limón 8d3640b90d oeqa: Move common files to oeqa/files instead of runtime only
Those files are used by runtime and sdk test cases, so move to
base directory of oeqa module.

[YOCTO #10599]

(From OE-Core rev: ec73e8a3d3149f3866b7bfc06f169c6e05e2d338)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:19 +00:00
Mariano Lopez bbb7366db3 oeqa/utils/__init__.py: Adds compatibility with bitbake logger
The bitbake logger changes the way debug is logged and adds
different levels within debug, this is passed as argument
to the function and breaks compatibility with vanilla loggers.

This implements a way to handle this adding a new function for
debug, that will dispatch the correct logging method signature.

Also overrides info method to use logging.INFO + 1 in order to
see plain data.

Also this commit fix the issue of not showing the test summary
and results when running from bitbake.

[YOCTO #10686]

(From OE-Core rev: 619c9ab308fbef9e3563dc661e432603e764b562)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:19 +00:00
Aníbal Limón 388503c032 oeqa/core: Add README
The README has an introduction and explains how to run the test suite
and creates a new Test component.

(From OE-Core rev: 9d474172c47695be1a61538f5b87ca8d9db25fa7)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:18 +00:00
Aníbal Limón 9a5f285fe3 oeqa/core/cases: Add example test cases
Serves as an first input of how to the OEQA framework works.

(From OE-Core rev: 115f80adf1b230c5d0392e7833e9aeb274642bcb)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:18 +00:00
Aníbal Limón 94cb20f20d oeqa/core/context: Add support of OETestContextExecutor
The OETestContextExecutor class supports to use oe-test for run core
test component also is a base class for the other test components
(runtime, sdk, selftest).

Te principal functionality is to support cmdline parsing and execution
of OETestContext, the test components could extend the common options
to provide specific ones. The common options between test components
are test data file, output log and test cases path's to scan.

Also it initializes the logger to be passed to the whole OEQA framework.

[YOCTO #10230]

(From OE-Core rev: 039deafa5f2c8fab31b8373b39f8bc219377b893)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:18 +00:00
Mariano Lopez 7bf63b28f1 oeqa/core: Add tests for the OEQA framework
This test suite covers the current functionality for the OEQA
framework.

For run certain test suite,

$ cd meta/lib/oeqa/core/tests
$ ./test_data.py

(From OE-Core rev: 7d7d0dc3736fc12ae7848de2785f0066e6470cd1)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:18 +00:00
Aníbal Limón 4da4091cee oeqa/core/decorator: Add support for OETestDataDepends and skipIfDataVar
The OETestDataDepends decorator skips a test case if a variable
isn't into test data (d).

The skipIfDataVar decorator skips a test case if a variable
has certain value.

(From OE-Core rev: 7dc519d20e835ee7693c31903e164c4bc0e5e598)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:18 +00:00
Mariano Lopez b685a7c1a8 oeqa/core/decorator: Add support for OETimeout decorator
The OETimeout provides support for specify certain timeout
in seconds for a test case, if the timeout is reach the SIGALRM
is sent and an exception is raised to notify the timeout.

[YOCTO #10235]

(From OE-Core rev: 1bf66a370361912e9950d7ff45e382c93622a169)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:18 +00:00
Mariano Lopez b8cebdb96c oeqa/core/decorator: Add support for OETestID and OETestTag
These two decorators stores certain TAG or ID for the test case
also provides support for filtering in loading step.

[YOCTO #10236]

(From OE-Core rev: 047af4ce864bbf98e2617b348ae9ccb77ac52871)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:18 +00:00
Aníbal Limón 95a2ec6aab oeqa/core/decorator: Add support for OETestDepends
The OETestDepends decorator could be used over test cases to
define some dependency between them.

At loading time sorting the tests to grauntee that a test case
executes before also raise an exception if found a circular
dependency between test cases.

At before test case run reviews if the dependency if meet, in the
case of don't it skips the test case run.

(From OE-Core rev: 2385bd3c8a7c012fd1cad5465ec7d34675552c75)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:18 +00:00
Aníbal Limón 13c8c08b95 oeqa/core: Add loader, context and decorator modules
loader: Implements OETestLoader handling OETestDecorator
and filtering support when load tests. The OETestLoader is
responsible to set custom methods, attrs of the OEQA
frameowork.

[YOCTO #10231]
[YOCTO #10317]
[YOCTO #10353]

decorator: Add base class OETestDecorator to provide a common
way to define decorators to be used over OETestCase's, every
decorator has a method to be called when loading tests and
before test execution starts. Special decorators could be
implemented for filter tests on loading phase.

context: Provides HIGH level API for loadTests and runTests
of certain test component (i.e. runtime, sdk, selftest).

[YOCTO #10230]

(From OE-Core rev: 275ef03b77ef5f23b75cb01c55206d1ab0261342)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:18 +00:00
Mariano Lopez abb55ab304 oeqa/core: Add utils module for OEQA framework
misc: Functions for transform object to other types.
path: Functions for path handling.
test: Functions for operations related to test cases and suites.

[YOCTO #10232]

(From OE-Core rev: 102d04ccca3ca89d41b76a8c44e0ca0f436b7004)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:18 +00:00
Aníbal Limón 08714d3b7e oeqa/core: Add base OEQA framework
case: Defines OETestCase base class that provides custom
    methods/attrs defined by the framework.
    Every OETestCase instance contains a reference to the test
    data (d), the test context (tc) and the logger.
    Also implements _oe{SetUp,TearDown}Class for make special
    handling of OEQA decorators and validations.

runner: Defines OETestRunner/OETestResult with support for RAW
    and XML result logs.

exception: Custom exceptions related to the OEQA framework based
    on class OEQAException.

[YOCTO #10230]
[YOCTO #10233]

(From OE-Core rev: c466086ccc4d4bb02d578a821cfb945945bfd529)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:18 +00:00
Ed Bartosh 253b8d7d58 selftest: fixed 3 wic test cases
Fixed test_systemd_bootdisk, test_wic_image_type and test_qemu
test cases by building core-image-minimal with correct configuration.

(From OE-Core rev: 897fe85d34302953c98d07ade4fa2dd749ae2d22)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:18 +00:00
Ed Bartosh 78d4d09cf2 selftest: wic: don't set WKS_FILE
Setting WKS_FILE variable should be done only when
wic image is expected to be built by bitbake.
If it's set for all images it breaks image building in
some cases.

(From OE-Core rev: 46b84310f7df157c2ef290a60f5c2136d4206f09)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:18 +00:00
Ed Bartosh 657f1e44b1 wic: fixed test_iso_image test case
Fixed isoimage-isohybrid plulgin and correspondent wic tet case:
- used wic-tools target when getting varlue of STAGING_LIBDIR variable
- ensured that image is built with efi and hddimg enabled

(From OE-Core rev: 5878484da64c38c7fde45bb06d76e22e608eb022)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:18 +00:00
Ed Bartosh dc4a612e60 image.bbclass: put .env files to ${STAGING_DIR}/imgdata/
As STAGING_DIR_TARGET started to point to a recipe specific
sysroot wic is not able to add .env files when .wks file refers
to multiple rootfs recipes.

Used STAGING_DIR instead of STAGING_DIR_TARGET to make the
directory with .env files the same for all recipes.

(From OE-Core rev: 3797cfd7473d3f9b7c0d999dcf9cd9608c8c7c6c)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:17 +00:00
Ed Bartosh 168f62e427 wic: fix getting path of native sysroot
wic used STAGING_DIR_NATIVE variable as a path to native sysroot.
This doesn't work with recipe specific sysroots as STAGING_DIR_NATIVE
points to the native sysroot of the current recipe.

Used RECIPE_SYSROOT_NATIVE variable of wic-tools recipe
to fix the issue.

(From OE-Core rev: de9d7d14cd03e4dfc5812890a53c79b706b56537)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:17 +00:00
Ed Bartosh a79b32c75a selftest: build wic-tools in the Wic.setUp
Built wic-tools recipe instead of set of tools recipes
to ensure that all tools are available from one recipe sysroot.

(From OE-Core rev: 52ce14d5ea667ffd319fdb3e147b6eb29505cf8e)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:17 +00:00
Richard Purdie 209a21ed46 selftest/devtool: Update to account for recipe specific sysroot
There is no common sysroot any more so add the libusb dependency using DEPENDS
and check for the output in the sysroot output directory so the tests
work with recipe specific sysroots.

(From OE-Core rev: ff30b833a423d300ec2b81bf80ef6733a6d8039b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:17 +00:00
Richard Purdie 9107d6ca14 Switch to Recipe Specific Sysroots
This patch is comparatively large and invasive. It does only do one thing, switching the
system to build using recipe specific sysroots and where changes could be isolated from it,
that has been done.

With the current single sysroot approach, its possible for software to find things which
aren't in their dependencies. This leads to a determinism problem and is a growing issue in
several of the market segments where OE makes sense. The way to solve this problem for OE is
to have seperate sysroots for each recipe and these will only contain the dependencies for
that recipe.

Its worth noting that this is not task specific sysroots and that OE's dependencies do vary
enormously by task. This did result in some implementation challenges. There is nothing stopping
the implementation of task specific sysroots at some later point based on this work but
that as deemed a bridge too far right now.

Implementation details:

* Rather than installing the sysroot artefacts into a combined sysroots, they are now placed in
  TMPDIR/sysroot-components/PACKAGE_ARCH/PN.

* WORKDIR/recipe-sysroot and WORKDIR/recipe-sysroot-native are built by hardlinking in files
  from the sysroot-component trees. These new directories are known as RECIPE_SYSROOT and
  RECIPE_SYSROOT_NATIVE.

* This construction is primarily done by a new do_prepare_recipe_sysroot task which runs
  before do_configure and consists of a call to the extend_recipe_sysroot function.

* Other tasks need things in the sysroot before/after this, e.g. do_patch needs quilt-native
  and do_package_write_deb needs dpkg-native. The code therefore inspects the dependencies
  for each task and adds extend_recipe_sysroot as a prefunc if it has populate_sysroot
  dependencies.

* We have to do a search/replace 'fixme' operation on the files installed into the sysroot to
  change hardcoded paths into the correct ones. We create a fixmepath file in the component
  directory which lists the files which need this operation.

* Some files have "postinstall" commands which need to run against them, e.g. gdk-pixbuf each
  time a new loader is added. These are handled by adding files in bindir with the name
  prefixed by "postinst-" and are run in each sysroot as its created if they're present.
  This did mean most sstate postinstalls have to be rewritten but there shouldn't be many of them.

* Since a recipe can have multiple tasks and these tasks can run against each other at the same
  time we have to have a lock when we perform write operations against the sysroot. We also have
  to maintain manifests of what we install against a task checksum of the dependency. If the
  checksum changes, we remove its files and then add the new ones.

* The autotools logic for filtering the view of m4 files is no longer needed (and was the model
  for the way extend_recipe_sysroot works).

* For autotools, we used to build a combined m4 macros directory which had both the native and
  target m4 files. We can no longer do this so we use the target sysroot as the default and add
  the native sysroot as an extra backup include path. If we don't do this, we'd have to build
  target pkg-config before we could built anything using pkg-config for example (ditto gettext).
  Such dependencies would be painful so we haven't required that.

* PKDDATA_DIR was moved out the sysroot and works as before using sstate to build a hybrid copy
  for each machine. The paths therefore changed, the behaviour did not.

* The ccache class had to be reworked to function with rss.

* The TCBOOTSTRAP sysroot for compiler bootstrap is no longer needed but the -initial data
  does have to be filtered out from the main recipe sysroots. Putting "-initial" in a normal
  recipe name therefore remains a bad idea.

* The logic in insane needed tweaks to deal with the new path layout, as did the debug source
  file extraction code in package.bbclass.

* The logic in sstate.bbclass had to be rewritten since it previously only performed search and
  replace on extracted sstate and we now need this to happen even if the compiled path was
  "correct". This in theory could cause a mild performance issue but since the sysroot data
  was the main data that needed this and we'd have to do it there regardless with rss, I've opted
  just to change the way the class for everything. The built output used to build the sstate output
  is now retained and installed rather than deleted.

* The search and replace logic used in sstate objects also seemed weak/incorrect and didn't hold
  up against testing. This has been rewritten too. There are some assumptions made about paths, we
  save the 'proper' search and replace operations to fixmepath.cmd but then ignore this. What is
  here works but is a little hardcoded and an area for future improvement.

* In order to work with eSDK we need a way to build something that looks like the old style sysroot.
  "bitbake build-sysroots" will construct such a sysroot based on everything in the components
  directory that matches the current MACHINE. It will allow transition of external tools and can
  built target or native variants or both. It also supports a clean task. I'd suggest not relying on
  this for anything other than transitional purposes though. To see XXX in that sysroot, you'd have
  to have built that in a previous bitbake invocation.

* pseudo is run out of its components directory. This is fine as its statically linked.

* The hacks for wayland to see allarch dependencies in the multilib case are no longer needed
  and can be dropped.

* wic needed more extensive changes to work with rss and the fixes are in a separate commit series

* Various oe-selftest tweaks were needed since tests did assume the location to binaries and the
  combined sysroot in several cases.

* Most missing dependencies this work found have been sent out as separate patches as they were found
  but a few tweaks are still included here.

* A late addition is that extend_recipe_sysroot became multilib aware and able to populate multilib
  sysroots. I had hoped not to have to add that complexity but the meta-environment recipe forced my
  hand. That implementation can probably be neater but this is on the list of things to cleanup later
  at this point.

In summary, the impact people will likely see after this change:

* Recipes may fail with missing dependencies, particularly native tools like gettext-native,
  glib-2.0-native and libxml2.0-native. Some hosts have these installed and will mask these errors

* Any recipe/class using SSTATEPOSTINSTFUNCS will need that code rewriting into a postinst

* There was a separate patch series dealing with roots postinst native dependency issues. Any postinst
  which expects native tools at rootfs time will need to mark that dependency with PACKAGE_WRITE_DEPS.

There could well be other issues. This has been tested repeatedly against our autobuilders and oe-selftest
and issues found have been fixed. We believe at least OE-Core is in good shape but that doesn't mean
we've found all the issues.

Also, the logging is a bit chatty at the moment. It does help if something goes wrong and goes to the
task logfiles, not the console so I've intentionally left this like that for now. We can turn it down
easily enough in due course.

(From OE-Core rev: 809746f56df4b91af014bf6a3f28997d6698ac78)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 12:05:17 +00:00
Paul Eggleton b1b59b8a4b oe-selftest: devtool: remove use of git -C
The -C option isn't available in versions of git older than 1.8.5,
and officially we only require git 1.8.3.1 or newer (and the latter is
the version you'll find on CentOS 7, so the test fails there). In any
case we can simply specify the working directory to runCmd() so just
do that instead.

(From OE-Core rev: ce5c7075d530c0950f2feed35f95fbcd9f50721f)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-20 11:53:51 +00:00
Ross Burton a304b94eae oeqa/selftest/devtool: rewrite modify testcase
The modify testcase had to be updated as it started failing when mdadm was
upgraded due to hardcoding version numbers in the test.  I then noticed how
inefficient the test was and mostly rewrote it.

Start by changing the minor modification to change "Linux Software RAID" (the
subtitle of the man page) to "antique pin sardine" (a nonsense phrase that is
unlikely to appear upstream), and neaten the logic.

Start by not removing sstate at the beginning of the test. To ensure builds
happen we can use -f and -C, and iterating the sstate cache is time consuming.

Don't bitbake mdadm repeatedly until it stabilizes, we can start with bitbake -C
unpack to ensure that a full build is done from scratch.

os.path.join has the interesting quirk that join(/foo, /bar) results in /bar, so
use oe.path.join instead of working around that manually.

Don't repeatedly call get_bb_var(), each call results in a call to bitbake.

These changes reduce the runtime of the test from over 600 seconds to around 160
seconds on my machine.

(From OE-Core rev: fc97963bc61bf16112859fe1d7e460a13d34baca)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-19 22:47:22 +00:00
Mariano Lopez 81c3d4824a oeqa/utils/qemurunner.py: Be sure to stop qemu-system
When runqemu fails, qemu-system process would keep running
and won't be killed, setpgrp() was used when runqemu was
a shell script but it seems it doesn't work always with python.

This would kill qemu-system explicity and to avoid leaving
it behind.

(From OE-Core rev: 9d2b1aa1bcfb2f1933a8eeb9470b4174d5da2f0d)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-19 22:47:22 +00:00
Mariano Lopez de17891185 oeqa/utils/qemurunner.py: Add missing sys module
This adds the missing sys module used by the child process
to exit. It seems the exception was cached in testimage and
selftest. It seems nobody noticed this because the module
is only used for sys.exit().

(From OE-Core rev: 66f66d1d763ff7bbaab9e8fcdf7fc882f2dfbb13)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-19 22:47:22 +00:00
Markus Lehtonen e530a95c5a oeqa.utils.metadata: include BB_NUMBER_THREADS and PARALLEL_MAKE
Inlude values of BB_NUMBER_THREADS and PARALLEL_MAKE in the metadata.

[YOCTO #10590]

(From OE-Core rev: 97c07a55815c2fc3915705317b6f30d212fa1d45)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-19 22:47:22 +00:00
Markus Lehtonen 56a7f41711 oeqa.utils.metadata: allow storing any bitbake config variables
Make it possible to store any bitbake config variables in the metadata.
Config values will be stored under a new config element in the xml report:
    <config>
        <variable name="MACHINE">qemux86</variable>
    </config>

The value of MACHINE is moved there instead of having a dedicated
<machine> element.

[YOCTO #10590]

(From OE-Core rev: 6e7e6e37664b0a86111272f5f6f4a4e1d0f23302)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-19 22:47:22 +00:00
Markus Lehtonen c74af56d4b oeqa.utils.metadata: add bitbake revision information
[YOCTO #10590]

(From OE-Core rev: 71ca7dab08fc500b231054249aacc90ae4aa85da)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-19 22:47:22 +00:00
Markus Lehtonen 47aac40869 oeqa.utils.metadata: have layer name as an attribute in xml
Have the layer name as an attribute instead of of the name of the
element itself. That is, have <layer name="layer_name"/> instead of
<layer_name/>. A bit better XML design.

[YOCTO #10590]

(From OE-Core rev: 50ea44c19005b536a2791113f8b536fd10548ead)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-19 22:47:22 +00:00
Markus Lehtonen 28858af49f oeqa.utils.metadata: add commit count information
Makes it easier to put the commits into a timeline.

[YOCTO #10590]

(From OE-Core rev: 7757970bced4ecd6503991c0cf11f4d9158f650c)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-19 22:47:22 +00:00
Markus Lehtonen ceed2c7eb5 oeqa.utils.metadata: rename 'revision' to 'commit'
Revision is a bit vague and could point to a tag, for example. Git
commit objects are unambiguous and persistent so be explicit that the
element should contain git commit hash.

[YOCTO #10590]

(From OE-Core rev: a6521b89e86ddba6bd646dd3974d7b0390323b56)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-19 22:47:22 +00:00
Markus Lehtonen 927e759bb2 oeqa.utils.metadata: fix retrieval of git branch and revision
Always return a valid branch name, or, '(nobranch)' if the current HEAD
is detached. Also, always return the hash of the commit object that HEAD
is pointing to. Previous code returned an incorrect branch name (or
crashed) e.g. in the case of detached HEAD.

[YOCTO #10590]

(From OE-Core rev: 02d3ba17a8090bd088beb973980651d664f713bb)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-19 22:47:22 +00:00
Markus Lehtonen 4e67232f34 oeqa.utils.metadata: drop 'unknown' git data elements
It's better just to not have the xml elements than to have elements with
faux data. One could have git branch named 'unknown', for example.

[YOCTO #10590]

(From OE-Core rev: fce531c21f5e56d0f416b3405a0b0fc5ba567679)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-19 22:47:22 +00:00
Markus Lehtonen df8c0540e4 oeqa.utils.metadata: re-organise distro information
Use the same format, based on /etc/os-release, as for host distro
information.

[YOCTO #10590]

(From OE-Core rev: 0156ef46ccf5334ee72f0202f1089249c62af37b)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-19 22:47:22 +00:00
Markus Lehtonen 8d51f62cd4 oeqa.utils.metadata: re-organise host distro information
Put all host distro data under one <host_distro> element. In addition
take the data directly from /etc/os-release instead of the "lsb API".
The /etc/os-release file is virtually ubiquitous, now, and using its
field names and values provides a more standardized and extensible
format.

[YOCTO #10590]

(From OE-Core rev: 98cad0b4063772dad94fea96edce1a5422256c32)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-19 22:47:22 +00:00
Jose Perez Carranza 2cd7613582 oeqa/selftest/tinfoil: add test IDs to tinfoil test cases
Testopia entrances were created and the IDs retrieved are
added to their corresponding test case on tinfoil script.

(From OE-Core rev: 486e5ad8270a4d3897f477ae5ae61422826f93ce)

Signed-off-by: Jose Perez Carranza <jose.perez.carranza@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-19 22:47:22 +00:00
Leonardo Sandoval 0bb40f0bb9 selftest: runtime-test: skip image-install test for poky-tiny
poky-tiny cannot build full-cmdline image, so skip this test in this case.

(From OE-Core rev: c2b1c562db160876fc3e1ee8b15bd07136d6ea7a)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-16 18:05:14 +00:00
Leonardo Sandoval 5c5db3a477 selftest: devtool: use distro agnostic recipes for devtool checks
The recipes being replaced are not compatible with all distros, so
use mraa and virtual/make for some checks.

(From OE-Core rev: bb32b232d686d1c057dede6f61ed4051b0088673)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-16 18:05:14 +00:00
Ross Burton 55228a1ca4 selftest/base: don't fetch DISTRO variable in constructor
Fetching the DISTRO variable in the base constructor means that we have to start
bitbake for every test case instance, which adds minutes to the startup time.

(From OE-Core rev: ac1c118dcb3cb27807b55115ef274a92bb512dd6)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-16 18:05:14 +00:00
Ross Burton 2abc37dbe8 selftest/bblayers: don't fetch a variable that is never used
(From OE-Core rev: f812f9518ef01abbdd9a6f415e5dc92263135f87)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-16 18:05:13 +00:00
Ross Burton cf8fcba8e7 selftest/archiver: don't build an image for a basic test
This test only exercises the include/exclude behaviour so it only needs to build
the two recipes that it tests against, not an entire image.

Part of #10874.

(From OE-Core rev: 9b02216be6c9dbf2f680db1ad1309bcb9fb32b23)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-16 18:05:13 +00:00
Chen Qi 988f2efd65 selftest/buildoptions.py: fix path assumption for DEPLOY_DIR_SRC
Fix path assumption for DEPLOY_DIR_SRC, otherwise, the testcase may fail
even if the functionality works well.

(From OE-Core rev: dab5d3901755a965cdd8f5b5e8ffb8e4cb79f2e5)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-16 18:05:13 +00:00
Chen Qi 0967a78689 selftest/bbtests.py: fix path assumption for LICENSE_DIRECTORY
Fix path assumption for LICENSE_DIRECTORY, otherwise, the test case
may fail even if the functionality it tests works well.

(From OE-Core rev: ae388652b8de0665390560e78429e10119d4d537)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-16 18:05:13 +00:00
Ola x Nilsson e97422f400 oe-selftest: devtool: Add test for externalsrc buildclean
Test both for S == B and S != B.

(From OE-Core rev: 3b46c1ac203717d85a1e2e8da067a69f066b7037)

Signed-off-by: Ola x Nilsson <olani@axis.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-16 18:05:13 +00:00
Chen Qi 9a1e8b9cf5 selftest/eSDK.py: fix sstate dir not found error
Fix the error below when SSTATE_DIR is not "${BUILDDIR}/sstate-cache".

  FileNotFoundError: [Errno 2] No such file or directory: '/xxx/../sstate-cache'

(From OE-Core rev: 785f0343d04c1684363b5289a3012cf7e1caa95f)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-16 18:05:12 +00:00
Jose Perez Carranza 970b76a728 runtime: Add cleanup for logrotate tests
Delete logrotate dir to avoid errors
when test are executed more than 1
time on the same target.

(From OE-Core rev: 02c98c848163a6837692fe74a91754701a44a6c0)

Signed-off-by: Jose Perez Carranza <jose.perez.carranza@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-16 18:05:12 +00:00
Randy Witt f64e321baf image_typedep.py: Add a test that ensures conversion type deps get added
Add a test that ensures if IMAGE_TYPEDEP_* contains a conversion type,
that the corresponding CONVERSION_DEPENDS_ for that type gets added to
the dependency tree for do_rootfs.

(From OE-Core rev: 9cf9c725f7d534c326ffd95ec539b041f4ad286f)

Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-16 18:05:12 +00:00
Ross Burton 949c7062b7 selftest/devtool: update test to work with new mtd-utils
The new mtd-utils (version 2.0) has been autotooled so the test needs to touch
Makefile.am instead of Makefile.

(From OE-Core rev: acce512a0b85853b5acf2ef07e4163a3b4f33a98)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-09 13:39:13 +00:00
Ola x Nilsson cfa6a7aa6f oe-selftest: devtool: Reverting a change should trigger rebuild
Add code to verify that not only does a change trigger a build, but so
does reverting that change.

Reverting a change in a devtool managed git repo may cause the current
checksum to match the checksum of a previous build, which will cause
bitbake to skip builds that are needed.

(From OE-Core rev: 58a31d8dd7293f14c70e56ec9639c420d15e7dfc)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-09 13:39:11 +00:00
Alejandro Hernandez 3f990e1b17 wic: Remove gummiboot test
Weve now migrated to systemd-boot, the gummiboot test on wic is no longer necessary

(From OE-Core rev: 0a8abc7681edec5f128ceb757559c5a50f139a9c)

Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-09 13:39:11 +00:00
Alejandro Hernandez 2dce2648e3 gummiboot: Remove/change gummiboot references with systemd-boot
After systemd-boot was introduced, its been tested for a while with no major
issues being found until now, this patch completely replaces all gummiboot
instances with systemd-boot ones, taking the next step into cleaning
up systemd-boot/gummiboot.

[YOCTO #10332]

(From OE-Core rev: f9a61d3400ad9068a6d83b8eb6aefe3098c58e68)

Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-09 13:39:11 +00:00
Richard Purdie 022a8b58c8 meta/scripts: Various getVar/getVarFlag expansion parameter fixes
There were a few straggling expansion parameter removals left for
getVar/getVarFlag where the odd whitespace meant they were missed
on previous passes. There were also some plain broken ussages such
as:

d.getVar('ALTERNATIVE_TARGET', old_name, True)
path = d.getVar('PATH', d, True)
d.getVar('IMAGE_ROOTFS', 'True')

which I've corrected (they happend to work by luck).

(From OE-Core rev: 688f7a64917a5ce5cbe12f8e5da4d47e265d240f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-09 13:39:11 +00:00
Jair Gonzalez dd37336b4d selftest/wic: extending test coverage for WIC script options
The previous WIC script selftest didn't cover all of its command line
options. Some option variants were included in existing test cases and
the following tests were added to complete covering them:

1552 Test wic --version
1553 Test wic help create
1554 Test wic help list
1555 Test wic list images
1556 Test wic list source-plugins
1557 Test wic listed images help
1558 Test debug
1563 Test skip build check
1564 Test build rootfs
1559 Test image vars directory selection
1562 Test alternate output directory

(From OE-Core rev: b4d52c3f1e0ad2c14028ff08c0938d1b24b7f648)

Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-22 08:50:17 +00:00
Jair Gonzalez f8745bb519 selftest/wic: reorganizing test methods by functionality
Part of the test methods were rearranged to group them by
functionality and identify more easily opportunities to extend
coverage.

(From OE-Core rev: f290a2c9b946e00dbc451592691596c656d49042)

Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-22 08:50:17 +00:00
Jair Gonzalez 285c2ed90e selftest/wic: code cleanup
The following changes were added to improve the code readability:

- Removed redundant backlashes between brackets
- Aligned continuation lines according to PEP8 style
- Refactored command execution in the next methods for enhanced
  legibility:

  - test_build_image_name(self)
  - test_gpt_image(self)
  - test_qemux86_directdisk(self)
  - test_rootfs_indirect_recipes(self)
  - test_iso_image(self)
  - test_mkgummidisk(self)
  - test_mkefidisk(self)
  - test_directdisk_bootloader_config(self)
  - test_qemu(self)
  - test_bmap(self)
  - test_systemd_bootdisk(self)
  - test_sdimage_bootpart(self)

(From OE-Core rev: 5ba21836fe3c061a6271f855a9e26c66004d330a)

Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-22 08:50:17 +00:00
Jair Gonzalez c48c0f38b1 selftest/wic: adding Testopia ID numbers to test cases missing it
The following test cases were assigned an ID number on Testopia:

1496 Test generation of .bmap file
1560 Test creation of systemd-bootdisk image
1561 Test creation of sdimage-bootpart image

(From OE-Core rev: b53e432206eaba7c6c67e4689c25b5e62d7ee9b2)

Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-22 08:50:17 +00:00
Maciej Borzecki 5903182484 oeqa/utils/commands.py: allow use of binaries from native sysroot
Tests may need to run a native tool that is not available on the host
filesystem, but can be built using one of the *-native recipes. In such case,
the tool will be available in native sysroot, and running in from that location
will require adjustments to PATH.

runCmd() can now take a path to native sysroot as one of its arguments and
setup PATH accordingly.

(From OE-Core rev: f2a04631949db72d4261d1c142c5044fad3741f9)

Signed-off-by: Maciej Borzecki <maciej.borzecki@rndity.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-22 08:50:16 +00:00
Richard Purdie 8ccf396c72 Revert "selftest/wic: extending test coverage for WIC script options"
This reverts commit 68cb3180c1b0dcee50812b21f98850d188d8621b as this wasn't
ready for merge and there are new better versions.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-20 17:06:38 +00:00
Ed Bartosh 2f4008d694 oe-selftest: import git module only when needed
git module is not included into standard Python
library and therefore causes import errors on the systems
where PythonGit is not installed.

As git module only used in the code implementing --repository
functionality it's better to import git only in the scope
that requires it.

[YOCTO #10821]

(From OE-Core rev: 66be32c1a075201d6ee0e9b9e10b84e6a2ace745)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-20 15:22:17 +00:00
Jair Gonzalez aa1c50a32b selftest/wic: extending test coverage for WIC script options
The previous WIC script selftest didn't cover all of its command
line options. The following test cases were added to complete
covering them:

1552 Test wic --version
1553 Test wic help create
1554 Test wic help list
1555 Test wic list images
1556 Test wic list source-plugins
1557 Test wic listed images help
1558 Test wic debug, skip-build-check and build_rootfs
1559 Test image vars directory selection
1562 Test alternate output directory

In addition, the following test cases were assigned an ID number on
Testopia:

1560 Test creation of systemd-bootdisk image
1561 Test creation of sdimage-bootpart image

Finally, part of the test methods were rearranged to group them by
functionality, and some cleanup was made to improve the code's
compliance with PEP8 style guide.

Fixes [YOCTO 10594]

(From OE-Core rev: 3d2ac67765020885a0996ebdd97a576ba37dbec0)

(From OE-Core rev: 68cb3180c1b0dcee50812b21f98850d188d8621b)

Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-20 15:22:16 +00:00
Joshua Lock c4e2c59088 meta: remove True option to getVar calls
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.

Search made with the following regex: getVar ?\(( ?[^,()]*), True\)

(From OE-Core rev: 7c552996597faaee2fbee185b250c0ee30ea3b5f)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:23 +00:00
Jose Perez Carranza 333890953d postinst: Add a test case to verify postinst scripts behavior
Add test case that verify behavior of postinst scripts at
roofts time and when is delayed to the first boot directly
on the target.

(From OE-Core rev: 82b171f3b37e6733997fc1e7685b7cac5a3476e7)

Signed-off-by: Jose Perez Carranza <jose.perez.carranza@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:23 +00:00
Francisco Pedraza 23d1c4ffb7 selftest: Test needed to verify postinst order
It verifies the following:

1. Compile a minimal image.
2. The compiled image will add the layer with the recipe postinst,
previously created at:
"meta-selftest/recipes-test"
3. Run QEMU.
4. Validate the task execution order.
[YOCTO #5319]

(From OE-Core rev: a8ff789a3bfedcbc4358db7907a45270d8b1b76a)

Signed-off-by: Francisco Pedraza <francisco.j.pedraza.gonzalez@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:23 +00:00
Ross Burton d535aba974 oeqa: move lib/oe tests to oe-selftest
These tests don't get ran often (as demonstrated by the fact that some were not
ported to Python 3), so move them to oeqa/selftest so they get executed
frequently and can be extended easily.

[ YOCTO #7376 ]

(From OE-Core rev: 2001979ad41e6fdd5a37b0f90a96708f39c9df07)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:22 +00:00
Leonardo Sandoval 030f62e9c8 selftest: buildoptions: skip read-only-image test depending on distro
Poky-tiny cannot build core-image-sato, so skip test (read-only-image)
in this case.

(From OE-Core rev: bcee8c614f28b38054f5d8c1c5251b3702cf113c)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:22 +00:00
Leonardo Sandoval e2ea2597c9 selftest: bblayers: remove linux kernel checks for show-recipes check
Preferred kernel recipes depends on the distro, so remove the kernel
checks to avoid failures on non-poky distros and make the test
distro agnostic.

(From OE-Core rev: ae92b72990b3ac804791b501d08126491fdddb7c)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:22 +00:00
Leonardo Sandoval 279c8ce310 selftest: bbtests: use minimal image so all distros can execute it
poky-tiny distro cannot build full-cmdline image, so use an image
(core-image-minimal) that can be built in all distros.

(From OE-Core rev: b293dd4200bbb3705c88af6113be7f43fbd0ed72)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:22 +00:00
Leonardo Sandoval 009adf73fa selftest: base: new object member to store the DISTRO value
Instead of quering it multiple times, query once and use it on
test method skip checks. Also, rename current distro sstate object
member to a more meaninful name.

(From OE-Core rev: bde9d99575a63ad2d7fd5974ce6ce19aad9a8984)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:22 +00:00
Mariano Lopez a92d27acd8 oeqa/utils/commands.py: Fix get_bb_vars() when called without arguments
Commit 9d55e9d489 broke calling get_bb_vars()
when called without arguments. This fix this issue.

(From OE-Core rev: 91f856426c7523e1ebdf6d6f93f5fa7e509d6e49)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 16:15:04 +00:00
Paul Eggleton 66adb6bdf7 oe-selftest: add basic tinfoil tests
Add some tests to verify that the new tinfoil API is operating
correctly.

(From OE-Core rev: 16afda66b861ba028c1152dcdcab2b7ebfbff965)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:30:50 +00:00
Paul Eggleton 0cba3cbcb4 oe-selftest: devtool: improve test_devtool_modify slightly
* Check that man .in file actually gets modified, since sed -i doesn't
  fail if it it doesn't
* Use a variable for man file path

(From OE-Core rev: 9ad36e945fa5b03726f78ba99e823eade8daa710)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:30:50 +00:00
Paul Eggleton 1f8cf63f8a oe-selftest: use tinfoil.parse_recipe()
Use tinfoil.parse_recipe() in order to allow oe-selftest to be used in
memres mode.

(From OE-Core rev: 499ee9006271112f22cfe08fa5ba5c21be95380b)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:30:49 +00:00
Paul Eggleton 93d675304d oe-selftest: make tinfoil quiet when using to start QEMU
We don't need to see the parsing/cache loading message in the
oe-selftest output, so use the newly added quiet option to disable it.

(From OE-Core rev: c023bc55ff000d1de891d1a8e2a163e94bf63de6)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:30:49 +00:00
Ola x Nilsson d65886bb69 oe-pkgdata-util: Make read-value handle override variables
Some variables in pkgdata files have a package-name override.  When
the bare variable can not be found, try with the override-variant.

PKGSIZE is one such variable, and already had special code to handle this.

Test included.

(From OE-Core rev: 6df99cda894033cba68bc6ab91e47f67e0d788a5)

Signed-off-by: Ola x Nilsson <ola.x.nilsson@axis.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:30:49 +00:00
California Sullivan 840ea755f2 parselogs.py: Don't clog QA with Joule errors
The Joule is very new hardware and there is ongoing kernel and firmware
work to fix these issues, which will be available in future kernel and
firmware releases. In the meantime, don't clog QA reports.

[YOCTO #10611]

(From OE-Core rev: facf9fa905100945738c13f9f79e938ed4a81030)

Signed-off-by: California Sullivan <california.l.sullivan@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-13 22:55:21 +00:00
Ola x Nilsson 2765231499 recipetool: selftest: Add test for recipetool plugin loading
Test that recipetool plugins are loaded in a well defined order.

(From OE-Core rev: 044de8424a454a7057906e44eb56e2134ebb17e4)

Signed-off-by: Ola x Nilsson <ola.x.nilsson@axis.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-13 22:55:20 +00:00
Ola x Nilsson 9170a88cbd devtool: selftest: add test for devtool plugin loading
Test that devtool plugins are loaded in a well defined order.

(From OE-Core rev: 0de81f0c8b29d8b442b3d099c3bec3fd345b6bfe)

Signed-off-by: Ola x Nilsson <ola.x.nilsson@axis.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-13 22:55:20 +00:00
Mariano Lopez 904b915369 oeqa/utils/metadata.py: Add metadata library
Adds functions to get metadata from the host running the tests.

[YOCTO #9954]

(From OE-Core rev: 10b05794254886e55c76f29f7778d783c550befa)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-13 22:55:19 +00:00
Mariano Lopez 9d55e9d489 oeqa/utils/commands.py: Make a copy of variables in get_bb_vars
The function get_bb_vars will remove items for the list passed
as the function argument, this will leave the caller with an
empty list and the function never says it will consume the items.

This hasn't been found before because only get_bb_var uses this
function.

(From OE-Core rev: 22b7fa24fefcc3974806d1b282c93b8c5880f6a4)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-13 22:55:19 +00:00
Juro Bystricky 90404e1f24 targetloader.py: drop test for ClassType
ClassType was removed from python3.
The code testing for ClassType kept throwing AttributeError exceptions:

    module 'types' has no attribute 'ClassType'

The exceptions prevented loading of any dynamically resolved target
controllers.

(From OE-Core rev: d62f18c39bc0ed3b0f5ac8465b393c15f2143ecf)

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12 15:16:42 +00:00
Ed Bartosh 1b32c6ed02 selftest: wic: fix test_qemu
Setting WKS_FILE variable in qemux86-64 made wic test to
use wrong wks file to produce an image and resulted in
test_qemu failure.

Used conditional assignment in qemux86-64 and explicitly
set WKS_FILE in wic testing suite to make the suite to use
wic-image-minimal.wsk. This should fix test_qemu failure.

(From OE-Core rev: 3bca4d18c2712e3b154bacfb917f0a749ebaddeb)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:38:06 +00:00
Jair Gonzalez 5956492c20 parselogs: Whitelist GPT warnings as the device is fully functional
The warning occurs when the GPT image is not the same size than the
media into which it's being flashed, causing the backup GPT table
not being at the end of the disk. However, this is expected as the
image is created before having the information about the destination
media. The error is harmless, so it will be whitelisted.

Fixes [YOCTO 10481].

(From OE-Core rev: 5cc5cdc788308a79f8f0706e6d794c602ef427ed)

Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:37:59 +00:00
California Sullivan 23d2a7d156 parselogs.py: Whitelist iwlwifi firmware load error messages
The iwlwifi module of any given kernel has a minimum and maximum
supported firmware version. The kernel begins by attempting to load the
maximum version, and decrements until it is successful. The 4.8 kernel's
maximum supported firmware version is 24, but thus far only 22 has been
released, meaning we get errors for 24 and 23.

Filter out iwlwifi firmware load error messages, as they are not
necessarily indicative of real problems.

(From OE-Core rev: 7df570c2310efac8f9898da15deaac2b7df16655)

Signed-off-by: California Sullivan <california.l.sullivan@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:37:59 +00:00
Jose Perez Carranza b81477d329 runtime: Update test cases numbers for runtime tests
Update test case numbers on runtime tests to do match
with templates defined on Testopia for 2.3 release

(From OE-Core rev: d9df762b4c62b74f6d3a1521642ea86c26793a22)

Signed-off-by: Jose Perez Carranza <jose.perez.carranza@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:08 +00:00
Michael Wood cfdc5b853e oe-selftest: toaster Remove redundant Toaster test
This test has been ported to be run as part of Toaster's own tests.

(From OE-Core rev: f6366781a34dbdb8ec2d73f4fb36359bc15c4f42)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:06 +00:00
California Sullivan f3c9e66e8d parselogs: Whitelist NUC6 firmware load error message on genericx86-64
This was already whitelisted, but the 4.8 kernel changed the error
message, causing it to get caught by parselogs again.

Fixes [YOCTO #10494].

(From OE-Core rev: e1bad14231115f3b1a2bf844ef5b2022c648b55d)

Signed-off-by: California Sullivan <california.l.sullivan@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:14 +00:00
Paul Eggleton 6e182aaa76 devtool: modify: support recipes with only local files as source
The hello-mod recipe is unusual in that it has only local files in
SRC_URI and builds these out of ${WORKDIR}. When you use devtool modify
on it, devtool puts all of those files in an "oe-local-files"
subdirectory of the source tree, which is not ${S} (or ${B}) any more
and thus building the recipe afterwards fails. It's a bit of a hack, but
symlink the files in oe-local-files into the source tree (and commit the
symlinks with an ignored commit so that the repo is clean) to work
around the problem. We only do this at time of extraction, so any files
added to or removed from oe-local-files after that won't be handled, but
I think there's a limit to how far we should go to support these kinds
of recipes - ultimately they are anomalies.

I initially tried a hacky workaround where I set effectively set B =
"${WORKDIR}" and that allowed it to build, but other things such as the
LIC_FILES_CHKSUM checks still broke because they expected to find files
in ${S}. Another hack where I set the sourcetree to point to the
oe-local-files subdirectory works for hello-mod but not for makedevs
since whilst that is similar, unlike hello-mod it does in fact have
files in the source tree (since it has a patch that adds COPYING) and
thus the same issue occurred.

Also tweak one of the tests that tries devtool modify / update-recipe on
the makedevs recipe to try building it since that would have caught this
issue.

Fixes [YOCTO #10616].

(From OE-Core rev: 857c06d6a1d161bf5a01311d07758bd4241929a3)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:14 +00:00
Paul Eggleton 39fba4d70f oe-selftest: devtool: test that updating a file with subdir= works
If you have a file:// entry in SRC_URI with a subdir= parameter that
makes it extract into the source tree, then when you update that file in
oe-local-files and run devtool update-recipe then you want the original
file to be updated. This was made to work by OE-Core commit
9069fef5dad5a873c8a8f720f7bcbc7625556309 together with
31f1bbad248c36a8c86dde4ff57ce42efc664082, however until now there was no
oe-selftest test to verify it.

Note that in order to succeed this test also requires the fix
"lib/oe/recipeutils: ignore archives by default in
get_recipe_local_files()" since the test recipe uses a local tarball.

(From OE-Core rev: 936eba3e1059d1dcd5e58c1ce76870fff7b11b3c)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:13 +00:00
Paul Eggleton 55a157f4e6 devtool: update-recipe: fix handling of compressed local patches
It is possible to use gzip or bzip2 to compress patches and still refer
to them in compressed form in the SRC_URI value within a recipe. If you
run "devtool modify" on such a recipe, make changes to the commit for
the patch and then run devtool update-recipe, we need to correctly
associate the commit back to the compressed patch file and re-compress
the patch, neither of which we were doing previously.

Additionally, add an oe-selftest test to ensure this doesn't regress in
future.

Fixes [YOCTO #8278].

(From OE-Core rev: e47d21624dfec6f71742b837e91da553f18a28c5)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:13 +00:00
Paul Eggleton e5a391795a oe-selftest: devtool: test update-recipe with only local files
Add a test to ensure devtool update-recipe works properly on recipes
that contain only local files (since the other tests we have didn't test
that).

Relates to [YOCTO #10563].

(From OE-Core rev: bdc844b3f0c3fbddcd3523095899a5bd29797704)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:12 +00:00
Ross Burton b654f5175d Revert "oeqa/selftest/kernel.py: Add new file destined for kernel related tests"
breaking on selftest

NOTE: recipe core-image-minimal-1.0-r0: task do_rootfs: Started
ERROR: core-image-minimal-1.0-r0 do_rootfs: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:license_create_manifest(d)
     0003:
File: '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/classes/license.bbclass', lineno: 48, function: license_create_manifest
     0044:    pkg_dic = {}
     0045:    for pkg in sorted(image_list_installed_packages(d)):
     0046:        pkg_info = os.path.join(d.getVar('PKGDATA_DIR', True),
     0047:                                'runtime-reverse', pkg)
 *** 0048:        pkg_name = os.path.basename(os.readlink(pkg_info))
     0049:
     0050:        pkg_dic[pkg_name] = oe.packagedata.read_pkgdatafile(pkg_info)
     0051:        if not "LICENSE" in pkg_dic[pkg_name].keys():
     0052:            pkg_lic_name = "LICENSE_" + pkg_name
Exception: FileNotFoundError: [Errno 2] No such file or directory: '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/sysroots/qemux86-64/pkgdata/runtime-reverse/kernel-4.8.3-yocto-standard'

This reverts commit c3d2df883a.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-15 15:19:52 +00:00
Paul Eggleton 43e652f3d1 devtool: add "rename" subcommand
When you run devtool add on a source tree we attempt to figure out the
correct name and version for the recipe. However, despite our best
efforts, sometimes the name and/or version we come up with isn't
correct, and the only way to remedy that up until now was to reset the
recipe, delete the source tree and start again, specifying the name this
time. To avoid this slightly painful procedure, add a "rename"
subcommand that lets you rename the recipe and/or change the version.

(From OE-Core rev: 9303d8055c45a0f6af295d70a6f6a8b9d8d8a7c9)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-07 11:04:22 +00:00
Paul Eggleton 7aebaa4204 oe-selftest: devtool: fix error message in _test_recipe_contents()
If a variable is being set in the recipe when we've explicitly passed
None as the value to _test_recipe_contents() indicating that it
shouldn't be set at all, then we should be printing out the variable
name in the assertion message but it seems like I forgot to do a
substitution. Also include the value for informational purposes.

(From OE-Core rev: 0dafcb158003fb13f82c266f607d9967fca321db)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-07 11:04:22 +00:00
Paul Eggleton e0b62c38ac recipetool: create: separate LICENSE items with & by default
recipetool sets the LICENSE value based on licenses detected from the
source tree. If there are multiple licenses then they were being
separated by spaces, but this isn't actually legal formatting and if
you're using "devtool add" you get a warning printed when devtool
parses the recipe internally.

Earlier I had made a conscious decision to do it this way since it's up
to the user to figure out whether the multiple licenses should all apply
(in which case they'd be separated with &) or if there is a choice of
license (in which case | is the correct separator). However, I've come
to the conclusion that we can just default to & and then the ugly
warning goes away, and it's the safest alternative of the two (and most
likely to be correct, since it's more common to have a codebase which is
made up of code with different licenses, i.e. all of them apply to the
combined work).

I've tweaked the comment that we add to the recipe to explicitly state
that we've used & and that the user needs to change that if that's not
accurate.

Fixes [YOCTO #10413].

(From OE-Core rev: ecac6aee8cf3313350b58c21012bcd67cfb915e4)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-07 11:04:22 +00:00
Costin Constantin c3d2df883a oeqa/selftest/kernel.py: Add new file destined for kernel related tests
[YP#7202]:  Test for linux-dummy
The new kernel.py file is intended for kernel related test cases.
The test for linux-dummy will ensure it is in good shape and can
be used as a kernel replacement at build time. To do this, the
test will first clean sstate for linux-dummy target, ensuring no
file is present in the stamps directory. After, core-image-minimal
is built, ensuring linux-dummy can be used as a kernel substitute.

(From OE-Core rev: 98c6ebf1e05158c689e01b785d32757847cdb10c)

Signed-off-by: Costin Constantin <costin.c.constantin@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-01 10:05:46 +00:00
Joshua Lock 751cb2c894 Remove RM_OLD_IMAGE, it's no longer useful
Since the move to put image deployment under sstate control in
d54339d4b1a7e884de636f6325ca60409ebd95ff old images are automatically
removed before a new image is deployed (the default behaviour of the
sstate logic).

RM_OLD_IMAGE is therefore no longer required to provide this
behaviour, remove the variable and its users.

(From OE-Core rev: 93631befe8b962bf99524746b49f4ebca336175c)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-15 10:01:42 +01:00
Richard Purdie 78c01995e3 oeqa/sshcontrol: Handle interrupted system call error
Deal with an interrupted system call gracefully:

|   File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-systemd/build/meta/lib/oeqa/utils/sshcontrol.py", line 55, in _run
|     if select.select([self.process.stdout], [], [], 5)[0] != []:
| InterruptedError: [Errno 4] Interrupted system call

(From OE-Core rev: 556125e4004fb7ac5169b59f51dc151f18c1806a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-07 16:43:58 +01:00
Ed Bartosh 09a9de45e1 wic: selftest: add test for sdimage-bootpart
Test creation of sdimage-bootpart image

(From OE-Core rev: 8b18568591f66aa9770798bc61123898de92b8a9)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-07 16:43:58 +01:00
Ed Bartosh bed4a13440 wic: selftest: add test for systemd-bootdisk
Test creation of systemd-bootdisk image.

(From OE-Core rev: b8eef88536b944ea35664b9a6a9496fb0a3f7988)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-07 16:43:58 +01:00
Markus Lehtonen b16001f3e2 oeqa.buildperf: measure apparent size instead of real disk usage
This change aligns disk usage measurements of the eSDK test with the old
build-perf-test.sh script. And thus, also makes the results between the
old and the new script comparable.

(From OE-Core rev: dadb84936b3672dcf07e5ab8226158136762801f)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-06 07:51:00 +01:00
California Sullivan ed12917d59 parselogs.py: Add disabling eDP error to x86_common whitelist
The NUC6 firmware tells the kernel to try and initialize an embedded
DisplayPort it does not have, causing this warning. Its harmless, so
just whitelist it.

Fixes [YOCTO #9434].

(From OE-Core rev: 4c3fb7f63aad4a5d1b9720c76091cd0646859c2a)

Signed-off-by: California Sullivan <california.l.sullivan@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-05 10:10:11 +01:00
Benjamin Esquivel d89e16d5cc oeqa/selftest: Update test after fetcher error changes
The following poky commit:

4359ef08 base.bbclass: Use bb.fatal() instead of raising FuncFailed

changed the way the fetcher error is reported.

Previous reporting:
...Function failed: Fetcher failure for URL:...

New reporting:
...Fetcher failure for URL:...

Updating how the check is done fixes the test error and accurately
confirms the tested scenario for test_invalid_recipe_src_uri.

[YOCTO #10370]

(From OE-Core rev: 197da17dc97cef87375ae9190c6d1495e1c615b9)

Signed-off-by: Benjamin Esquivel <benjamin.esquivel@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-05 10:10:10 +01:00
Richard Purdie 2d7df9be8d oeqa/sstatetests: Add test for multilib allarch checksums
Switching between multilib configurations should not change allarch recipe
or nativesdk checksums. Add a new sstate test for this based on the standard
allarch test.

(From OE-Core rev: 660543601171f88c75fb4e90f34dac86037f3f23)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-01 21:45:57 +01:00
Francisco Pedraza 7ae326c76b oeqa/utils: Add StreamHandler to logger
StreamHandler was added due missing log information on the console in
oe-selftest with Qemu Runner

(From OE-Core rev: a4e2df151af781edbcb6b0e17b51b5ed226bf77f)

Signed-off-by: Francisco Pedraza <francisco.j.pedraza.gonzalez@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-01 21:45:57 +01:00
Jianxun Zhang e0e8cd039c parselogs: Update uvesafb errors in qemu whitelist
This change only whitelists the timeout message in infinite
wait case in uvesafb driver.

With the latest timeout patch introducing infinite wait in
uvesafb driver, we whitelist the timeout message since it works
as a warning for issues related to timeout not to be fixed in
build servers.

We remove other errors for bug-discovering purposes in some cases
where these lines are still worthy to be caught (not whitelisted):

The removed errors show up again in the infinite wait case. It
indicates a different root cause or the timeout patch doesn't work
correctly.

Timeout happens when developers explicitly set a non-negative timeout
of a limited period to wait for task completion in uvesafb driver.

Timeout or/and errors occur when kernel doesn't have the latest
timeout patch in driver.

Note: The latest timeout patch is tracked by:
a2966330bc

[YOCTO #8245]

(From OE-Core rev: 2e15b478343c6703c37b9a45e61c9de200d98027)

Signed-off-by: Jianxun Zhang <jianxun.zhang@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-30 16:51:15 +01:00
Richard Purdie 12905346a0 oeqa/sstatetests: Ensure we cover deb packaging backend for sstate test
Currently we weren't testing the deb backaned for sstate correctness
and there was a bug that had crept in. Ensure we cover all package
backends with the test regardless of what the distro/conf sets.

(From OE-Core rev: cdaafc3729700778d95afc2413553d7b41c1317b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-28 10:16:04 +01:00
Jussi Kukkonen 46c5e7adb8 lib/oeqa: Regenerate galculator configure
galculator configure seems to be so old it does not
recognise --with-libtool-sysroot: regenerate configure.

Fixes [YOCTO #10191].

(From OE-Core rev: 3e838773462e77cb2e3ba9a69534260f89ca4904)

Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-28 10:16:04 +01:00
California Sullivan bb744b82c7 parselogs.py: Add amd_nb error to x86_common whitelist
This has always silently failed on hardware without AMD Northbridge,
and a recent kernel patch made it not silent. It would be ideal to only
whitelist the error for genericx86 MACHINEs and disable the CONFIG
option that enables it in intel-* MACHINEs, but in order to disable
this configuration option we would have to enable EXPERT and
DEBUG_KERNEL, which we don't want. Instead just whitelist it on all
x86 MACHINEs.

Fixes [YOCTO #10261].

(From OE-Core rev: 9c432dae1045a087f8eb2de7c9bd3a9cbd46c459)

Signed-off-by: California Sullivan <california.l.sullivan@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-28 10:16:03 +01:00
Jose Lamego c902e7b937 oeqa/selftest/base: backup and restore local configuration files
Selftests' cleanup method during test setup is not capable of
restoring local configuration files that remain modified after
aborting a test through a keyboard interruption.
This change creates backups for local.conf and bblayers.conf at
test setup, restore them when found, and deletes them at cleanup.

[YOCTO #9390]

(From OE-Core rev: 0877278e07e4c2494c4c23199490dc47a5cee69d)

Signed-off-by: Jose Lamego <jose.a.lamego@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-23 14:56:39 +01:00
Ross Burton a767c78abe oeqa/selftest/lic-checksum: don't report the expected failure to errors.yp
This test has a bitbake invocation that is expected to fail, so inhibit
report-error running.

(From OE-Core rev: b2771e17a5f301423f65be9f93c9c1b1e7f8ab93)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-20 15:11:09 +01:00
Aníbal Limón a8cf594bce oeqa/runtime/parselogs.py: Add ignore of tsc calibration fail in x86
We are experimenting failures to calibrate CPU's using TSC in x86
VM's due to usage of nested KVM [1], this is a known issue [2][3]
in virtualization environments, for detail explnation see [4].

Also we already have an ignore for 'TSC Fast calibration fail'.

[1] http://errors.yoctoproject.org/Errors/Details/83684/
[2] https://bugzilla.redhat.com/show_bug.cgi?id=814231
[3] https://lists.nongnu.org/archive/html/qemu-devel/2010-09/msg01547.html
[4] https://www.kernel.org/doc/Documentation/virtual/kvm/timekeeping.txt

(From OE-Core rev: 2271f59a0f506f89f9fea6777701c4b40790ddd9)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-20 15:11:09 +01:00
Paul Eggleton 7ba62d65d2 oeqa/sdkext/devtool: use a smaller module to test node.js functionality
The "forever" package, despite its innocent description, actually drags
in a surprising number of dependencies and as a result the nodejs test
takes up to 10 minutes as a result. Pick a different example with a much
more reasonable set of dependencies.

Addresses part of [YOCTO #10254].

(From OE-Core rev: 638ee71da967f093071ba25e0ea5c467dab65339)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-16 15:24:02 +01:00
bavery 92862c678f qemurunner: print out the runqemu command
This logs the launch command that was used for runqemu while running -c
testimage.  This way, if I'd like to easily launch qemu manually in
order to debug a failed test, I know what commmand was run to create
the qemu instance.

(From OE-Core rev: 34aa20c6f323bbf7ad53beb643126e4e03634708)

Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-16 15:24:02 +01:00
Ross Burton a7a6d05579 oeqa/oetest: show stderr when running commands
To help debug failures, redirect stderr to stdout in oeSDKTest.run() and
oeSDKExtTest.run().

(From OE-Core rev: 4cd143e0de7f0082f60f273f442f6255f28ec3e7)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-16 15:24:02 +01:00
Aníbal Limón 62c45ffbc7 oeqa/utils/decorators: LogResults fix race condition in linkfile
In order to avoid race condition when test if exists the linkfile
use bb.utils.lock, the best solution is to create a unique name
for the link file.

There is no way to create a unique linkfile name at this decorator
because is needed the machine and image variables, those variables
can't be passed easily in this code.

To avoid broke test export functionality use a try/except because bb
isn't available when use test export

[YOCTO #10225]

(From OE-Core rev: 059d475b6bce1e5414170a4fe2e7989f6b0eacd6)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-15 12:15:07 +01:00
Richard Purdie 56a4900273 Revert "oeqa.runtime.smart: work around smart race issues"
We have a proper fix now so we can remove the workaround.

This reverts commit 4d268abc2fc892c5d34449f78c8e9f2b1a9d6bac.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-14 22:22:13 +01:00
Edwin Plauchu 2c7849fdcb oeqa: Remove linux user utilized for rpm test.
When trying to re-test smart rpm tests. A fail arises
due to a linux user previously created upon the image.
We've added a few lines to delete such user and his home dir
when finishing test.

[YOCTO #9204]

(From OE-Core rev: d1a80ac434bb798634bbb792ff7df59148ec26be)

Signed-off-by: Edwin Plauchu <edwin.plauchu.camacho@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-14 22:22:07 +01:00
Markus Lehtonen 0736079dc6 oeqa.buildperf: another fix for splitting 'nevr' string
When processing buildstats we determine recipe name, epoch, version and
revision from the per-recipe buildstat directory name.  One previous
patch made an assumption that package version starts with a number.
That might not be true because of a packaging mistake or whatever
reason. Thus, if a version starting with a number is not found, fall
back to the "old" method of just taking the second-last dash-delimited
part (the one before revision).

(From OE-Core rev: 936eb9aac055f1d4f41bb351477a0f5bebf76a0e)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-14 22:22:07 +01:00
Richard Purdie 996aaf93ff oeqa/parselogs: Don't use cwd for file transfers
If you run:

MACHINE=A bitbake <image> -c testimage
MACHINE=B bitbake <image> -c testimage

and A has errors in parselogs, machine B can pick these up and cause
immense confusion. This is because the test transfers the log files
to cwd which is usually TOPDIR. This is clearly bad and this patch
uses a subdir of WORKDIR to ensure machines don't contaminate each
other.

Also ensure any previous logs are cleaned up from any existing
transfer directory.

(From OE-Core rev: ac8f1e58ca3a0945795087cad9443be3e3e6ead8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-14 22:22:06 +01:00
Markus Lehtonen 68e35c8e37 oeqa.runtime.smart: work around smart race issues
Yucku hack around test failures which ultimately are caused by a race in
smartpm itself. Issuing smartpm commands in quick succession causes
races in package cache of smartpm on some systems. This patch mitigates
the problem by sleeping for 1 second after each smartpm command that
modifies the system.

[YOCTO #10244]

(From OE-Core rev: 4d268abc2fc892c5d34449f78c8e9f2b1a9d6bac)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-09 12:12:23 +01:00
Richard Purdie d3991342ed oeqa/runtime/smart: Prune feeds to save memory
Full package feed indexes overload a 256MB image so reduce the number of rpms
the feed. Filter to p* since we use the psplash packages and this leaves some
allarch and machine arch packages too.

[YOCTO #8771]

(From OE-Core rev: f352c3b71cbf50846c7de31046202296b38713cc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-09 12:12:23 +01:00
Richard Purdie 8c46605d16 oeqa: Use snapshot instead of copying the rootfs image
Rather than copying images, use the snapshot option to qemu. This fixes a regression
caused by the recent runqemu changes where the wrong images were being testes since
the image is copied without the qemuboot.conf file. This means the latest image is
found by runqemu rather than the specified one, leading to various confused testing
results.

It could be fixed by copying more files but use snapshot mode instead.

(From OE-Core rev: eab91997d415b0e690b3482749a32087e6a8b00a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-09 12:07:32 +01:00
Robert Yang 95331c6d4e qemurunner.py/qemutinyrunner.py: remove runqemu-internal
There is no runqemu-internal any more.

(From OE-Core rev: 14bacf7203ab7a638b67eb143225d8c75bbb703d)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-09 12:07:32 +01:00
Robert Yang af7ec7afc0 parselogs: Whitelist qemux86 error message with qemu 2.7.0
qemu 2.7.0 introduces kernel errors:

[    2.310768] pci 0000:00:00.0: [11ab:4620] type 00 class 0x060000
[    2.311338] pci 0000:00:00.0: [Firmware Bug]: reg 0x14: invalid BAR (can't size)
[    2.311604] pci 0000:00:00.0: [Firmware Bug]: reg 0x18: invalid BAR (can't size)
[    2.311835] pci 0000:00:00.0: [Firmware Bug]: reg 0x1c: invalid BAR (can't size)
[    2.312063] pci 0000:00:00.0: [Firmware Bug]: reg 0x20: invalid BAR (can't size)
[    2.312323] pci 0000:00:00.0: [Firmware Bug]: reg 0x24: invalid BAR (can't size)
[    2.314320] pci 0000:00:0a.0: [8086:7110] type 00 class 0x060100
[    2.315363] pci 0000:00:0a.1: [8086:7111] type 00 class 0x010180

Whitelist this for now since this is preferable to the random failures
we're seeing from qemuppc with 2.6.0.

(From OE-Core rev: 4d542cdc86c34f0f4a3dde8b0aab059bca76a9fb)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-09 11:53:37 +01:00
Richard Purdie e399ca1dd8 parselogs: Whitelist qemuppc error message with qemu 2.7.0
qemu 2.7.0 introduces kernel errors:

[0.474981] pci 0000:00:0f.0: reg 0x10: [io  0x0400-0x041f]
[0.483796] pci 0000:00:0f.0: reg 0x20: can't handle BAR above 4GB (bus address 0xffffffff82800000)
[0.484204] pci 0000:00:0f.0: reg 0x20: [mem size 0x00800000 64bit pref]
[0.488077] pci 0000:00:0f.0: reg 0x30: [mem 0x83000000-0x8303ffff pref]
[0.488903] pci 0000:00:10.0: [1af4:1005] type 00 class 0x00ff00
[0.490485] pci 0000:00:10.0: reg 0x10: [io  0x0480-0x049f]
[0.496512] pci 0000:00:10.0: reg 0x20: can't handle BAR above 4GB (bus address 0xffffffff83800000)
[0.496783] pci 0000:00:10.0: reg 0x20: [mem size 0x00800000 64bit pref]
[0.500345] pci 0000:00:11.0: [1af4:1001] type 00 class 0x010000
[0.501790] pci 0000:00:11.0: reg 0x10: [io  0x0500-0x053f]
[0.507362] pci 0000:00:11.0: reg 0x20: can't handle BAR above 4GB (bus address 0xffffffff84000000)
[0.507677] pci 0000:00:11.0: reg 0x20: [mem size 0x00800000 64bit pref]
[0.513905] pci_bus 0000:00: busn_res: [bus 00-ff] end is updated to 00
[0.516493] PCI 0000:00 Cannot reserve Legacy IO [io  0x0000-0x0fff]
[0.517512] pci 0000:00:0f.0: BAR 4: assigned [mem 0x80800000-0x80ffffff 64bit pref]
[0.518877] pci 0000:00:10.0: BAR 4: assigned [mem 0x82800000-0x82ffffff 64bit pref]
[0.519890] pci 0000:00:11.0: BAR 4: assigned [mem 0x83800000-0x83ffffff 64bit pref]

Whitelist this for now since this is preferable to the random failures
we're seeing from qemuppc with 2.6.0.

(From OE-Core rev: 2472ed5393ab01ad79c011ea19c73224ed5125de)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-09 11:53:37 +01:00
Markus Lehtonen 98264a9191 oeqa.buildperf: be sure to use the latest buildstats
Be sure to take the latest buildstats if multiple buildstats are found.

(From OE-Core rev: bad495f0d0144728a0132c3d3c4d98c24ead4afd)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-08 00:32:43 +01:00
Markus Lehtonen d021889ba9 oeqa.buildperf: try harder when splitting 'nevr' string
Try to be more intelligent when splitting out recipe name, epoch,
version and revision from the buildstat directory name. Previous
assumption was that package versions never contain a dash but obviously
that is not necessarily true. The new assumption is that the package
version starts with a number.

(From OE-Core rev: 91d3fce1eb3e27d646afba8cf3c03ae560412d1d)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-08 00:32:43 +01:00
Markus Lehtonen f7ca989ddc oeqa.buildperf: correct globalres time format
Always use two digits for (integer part of) seconds, i.e. show '1:02.34'
instead of '1:2.34'.

(From OE-Core rev: 55bb6816aca39bfa25d4f7e2158a57a5f0ac1cca)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-06 10:24:04 +01:00
Joshua Lock e8e81789f9 selftest/liboe: add a test for copyhardlinktree()
Add a simple test to validate that the number of files in the
destination matches the number of files in the source after the
copyhardlinktree() has been performed.

(From OE-Core rev: ca5c718b309524e46818627f8b5c9260d009472d)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-06 10:24:04 +01:00
Richard Purdie 42e2b97441 oeqa/parselogs: Add qemuarm64 warning from 4.8 kernel to whitelist
(From OE-Core rev: ae865fee26d2a32ae07236fc7aa1cf1b234a2156)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-05 11:56:01 +01:00
Bruce Ashfield d9c2c02e96 oeqa/parselogs/qemuarm: Whitelist amba and jitter for 4.8+ kernels
With the update to the 4.8 kernel the versatile platform (and hence
qemuarm) has switched to a device tree boot.

We are using an ummodified mainline kernel versatilepb device tree,
which includes definitions of multiple amba devices. These devices
are not present in the qemu system emulation, hence throw warnings
during boot.

These warnings are not unique to oe-core, and rather than carry kernel
patches to the device tree (for now), we whitelist the known warnings
so qa testing will pass. We also can't turn amba off completely, since
it is providing valid devices (like the serial port) and AMBA is
force selected by other kconfig values.

We also have a jitterentropy warning that shows up on some hosts.
This warning is harmless, and like amba we can't turn it off in a
fragment since it is force selected by crypto (and we'd rather not
turn all crypto off). So we add it to the whitelist while investigations
continue into what is needed in the host to support this fully.

(From OE-Core rev: f5315b8c7998611da9984fd6bce2b48d6304ff6c)

Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-05 11:56:01 +01:00
Ed Bartosh 9cc4492732 image: Deploy images to IMGDEPLOYDIR
Changed deployment directory from DEPLOY_DIR_IMAGE to
IMGDEPLOYDIR to make sstate machinery to do final deployment and
generate manifest.

Renamed variable deploy_dir to deploy_dir_image in selftest code
to avoid confusion with DEPLOYDIR variable.

Updated the code of rootfs.py:Rootfs class to use IMGDEPLOYDIR variable
as it's now used as a new deployment destination.

(From OE-Core rev: 6d969bacc718e21a5246d4da9bf9639dcae29b02)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-04 00:07:28 +01:00
Francisco Pedraza 0e38688bca oeqa/selftest Adds eSDK test cases to devtool verification.
The covered functions are, install libraries headers and image generation
binary feeds.

(From OE-Core rev: 994f8a41a16d0b82a1f7dfbcbbcc1df08225b14e)

Signed-off-by: Francisco Pedraza <francisco.j.pedraza.gonzalez@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 23:45:54 +01:00
Francisco Pedraza 1da953d631 /oeqa/sdkext Adds verification for devtool on eSDK.
The covered funcions are, build make, build esdk package, build cmake
extend autotools recipe creation, kernel module,
node.js installation and recipe creation.

(From OE-Core rev: 574a5d4cf3e79815aecc4d198545119d3bbfb023)

Signed-off-by: Francisco Pedraza <francisco.j.pedraza.gonzalez@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 23:45:54 +01:00
Paul Eggleton e5f61f85c5 oe-selftest: devtool: fix test after recent change
OE-Core commit d3057cba0b01484712fcee3c52373c143608a436 fixed handling
of wildcard bbappends, which means that this test's expectations about
the bbappend file name are no longer met. devtool finish is meant to use
wildcard bbappends so fix the test accordingly.

(From OE-Core rev: 21603566e4a2e709dcb4a940b49d870c91c822be)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 23:45:52 +01:00
Joshua Lock 1429d9d463 oeqa.selftest.liboe: add test for xattr in copytree
Add a test to ensure that oe.path.copytree() preserves extended
attributes on files.

(From OE-Core rev: 2b047b8e3218f95978e41fee13635bff9af03dd6)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:42 +01:00
Joshua Lock 97677c11cf oeqa.selftest: add a test for oe.path.copytree()
One motivation for the use of cpio in oe.path.copytree() was to
ensure that files with spaces in their names were copied. Add a new
unittest module to test the OE module with a test case for copytree
with a spaces in a filename.

(From OE-Core rev: a408f8310d9426db4439cf8db0cf49f9bfe90b3b)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:42 +01:00
Markus Lehtonen 7d77c02401 oeqa.buildperf: include commands log file name in results.json
(From OE-Core rev: b22a71cf3a53a33763ff02608119d2c73cbde006)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:42 +01:00
Markus Lehtonen c39db4bc45 oeqa.buildperf: include buildstats file name in results.json
No need to do lsdir magic for finding buildstats when reading results.

(From OE-Core rev: 4502f0979bf2e8698bb196345b89b170641fd43f)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:42 +01:00
Markus Lehtonen c5d1301245 oeqa.buildperf: show skipped tests in results, too
(From OE-Core rev: 4112779f9f314148b475fc4b8e33146de8be6b27)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:42 +01:00
Markus Lehtonen 81b8ccc1f6 oeqa.buildperf: convert buildstats into json format
Instead of archiving buildstats in raw text file format convert all
buildstats into one json-formatted file. Some redundant information,
i.e. 'Event:', 'utime:', 'stime:', 'cutime:' and 'cstime:' fields, are
dropped.

(From OE-Core rev: efcf74b194f2a40eb3e6359dd41386db3eb25287)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:42 +01:00
Markus Lehtonen f1fb013d48 oeqa.buildperf: measure io stat
Add data from /proc/<pid>/io to system resource measurements.

(From OE-Core rev: e69a46a77854fac1169a09e0c5b70fa4b972255a)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:42 +01:00
Markus Lehtonen 0b332039ea oeqa.buildperf: don't use Gnu time
Use Python standard library functionality instead of the time utility
for measuring elapsed (wall clock) time of commands. The time.* log
files are also ditched. However, the same detailed resource usage data,
previously found in time.* logs is now provided in results.json file.
This data is collected through the resource module of Python.

(From OE-Core rev: d5ad818dd501b18379aa3540bffa9b920d7c3bab)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:42 +01:00
Markus Lehtonen 33a38bc18a oeqa.buildperf: rename buildstats directories
Change directory name from 'buildstats-<test_name>' to just
'buildstats'. However, this patch adds the possibility to label
buildstats directory name with a postfix which makes it possible to save
multiple buildstats per test, for example.

(From OE-Core rev: 8997556040b2e7bfcfa6a75d4d97eb2e32207217)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:41 +01:00
Markus Lehtonen 6d75f39f09 oeqa.buildperf: separate output dir for each test
Store the output data of each test in an individual subdirectory instead
of storing everything in the root output directory.

(From OE-Core rev: 64ff34df96aa9a74dd4303f76ec711aa5e9d5030)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:41 +01:00
Markus Lehtonen 700ebe996a oeqa.buildperf: strip date from buildstats directory path
Archive buildstats in a directory like 'buildstats' instead of something
like 'buildstats/20160513120000'.

(From OE-Core rev: 95138cdc70bb7f9b7ab74e1d83305f009790dccc)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:41 +01:00
Markus Lehtonen e8c47a6343 oeqa.buildperf: enable json-formatted results
Automatically create a json.formatted file (results.json) in the results
directory that contains results from all tests.

(From OE-Core rev: 6df3263531a41805b2280bb999cb4a73f9f91eae)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:41 +01:00
Markus Lehtonen 44188933ce oeqa.buildperf: add 'product' to test result data
This defaults to 'oe-core' but can be defined using the
OE_BUILDPERF_PRODUCT environment variable.

(From OE-Core rev: a22cc3e04001be5d11bd85dbdceb7088cae7c735)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:41 +01:00
Markus Lehtonen 899b17413c oeqa.buildperf: treat failed measurements as errors
Now failed measurements correctly cause a test failure (recorded as an
error). There should be no need to continue the test if one step fails,
especially now that the tests don't depend on each other.

(From OE-Core rev: 446e32aadc775ca146d12173b1463f524d7fe6ef)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:41 +01:00
Markus Lehtonen 85b7b10b4a oeqa.buildperf: make tests independent
Add test set-up functionality so that the individual tests do not depend
on each other. This should make sure that a failure in one test does not
affect the results of another test. The patch also makes it reasonable
to run only a subset of the tests by using the --run-tests option.

The increase in total execution time of the full suite - caused by the
additional set-up steps - is insignificant because normally no
additional tasks need to be run. The previous test has already done all
set-up work.

(From OE-Core rev: 69b3c63e32d09ea4a41b21daacdff6bf1fc447c1)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:41 +01:00
Markus Lehtonen 6722b0412c oeqa.buildperf: fix checking of invalid results
The test status check done when writing globalres log was incorrect.

(From OE-Core rev: 3efbd49fd80d2b349a8fd44dbcd509168dbc1061)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:41 +01:00
Markus Lehtonen 7155a9b64d oe-build-perf-test: add {git_commit_count} keyword for --commit-results-tag
Makes it possible to create easily sortable tags. Also, the default tag
format is updated to use the new keyword.

(From OE-Core rev: e3161654d75dfc3b059c519205b38b26e3ffb215)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-25 23:03:47 +01:00
Markus Lehtonen d0bac259bd oeqa.buildperf: add git commit count to result data
This number represents the number of commits since the beginning of git
history until the tested revision. This helps e.g. in ordering results.

(From OE-Core rev: b52070dd057ff5b410cd193f9be2f25bc4c506cc)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-25 23:03:47 +01:00
Markus Lehtonen caf6ad889c oe-build-perf-test: new {tag_num} keyword for --commit-results-tag
This makes it possible to create numbered tags, where the "basename" of
the tag is the same and the only difference is an (automatically)
increasing index number. This is useful if you do multiple test runs on
the same commit. For example, using:
--commit-results-tag {tester_host}/{git_commit}/{tag_num}

would give you tags something like:
myhost/decb3119dffd3fd38b800bebc1e510f9217a152e/0
myhost/decb3119dffd3fd38b800bebc1e510f9217a152e/1
...

The default tag format is updated to use this new keyword in order to
prevent unintentional tag name clashes.

(From OE-Core rev: cf2aba16338a147f81802f48d2e24a96c7133548)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-25 23:03:47 +01:00
Markus Lehtonen 06b2c75c7e oe-build-perf-test: tag results committed to Git
Create a Git tag when committing results to a Git repository. This patch
also implements --commit-results-tag command line option for controlling
the tag name. The value
is a format string where the following fields may be used:
- {git_branch} - target branch being tested
- {git_commit} - target commit being tested
- {tester_host} - hostname of the tester machine

Tagging can be disabled by giving an empty string to
--commit-results-tag. The option has no effect if --commit-results is
not defined.

(From OE-Core rev: 60059ff5b81d6ba9ba344161d51d1290559ac2df)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-25 23:03:47 +01:00
Markus Lehtonen 6d9c52fe4b oeqa.utils.git: implement init() method
Method for doing 'git init'.

(From OE-Core rev: c848e1dac68cd859a563a82286f8bc5ddabaa423)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-25 23:03:47 +01:00
Markus Lehtonen 96e8337830 oe-build-perf-test: implement --commit-results-branch
A new command line option for defining the branch where results are
commited. The value is actually a format string accepting two field
names:
- {git_branch} expands to the name of the target branch being tested
- {tester_host} expands to the hostname of the tester machine

The option has no effect if --commit-results is not used.

(From OE-Core rev: b54b63395ec632748a57a702812c8a9a07af35ab)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-25 23:03:47 +01:00
Markus Lehtonen 8335422b00 oe-build-perf-test: support committing results data to Git
Implement a new command line option '--commit-results' which commits the
test results data into a Git repository. The given path must be an
existing initialized local Git repository.

(From OE-Core rev: b6f635513ca971402e7a970acc2168fb5d4a9476)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-25 23:03:47 +01:00
Markus Lehtonen 66540ae5c1 oeqa.buildperf: use term commit instead of revision
This is basically a internal change, at this point. Term 'commit' better
represents the data we actually have. Term 'revision' is more vague and
could be understood to point to a tag object, for example.

(From OE-Core rev: f49cf7959b8aaa52b79b22a5884c6aa580a50302)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-25 23:03:47 +01:00
Markus Lehtonen 979c735678 oeqa.utils.git.GitRepo: new arg to require topdir
Add a new 'is_topdir' argument to the GitRepo init method which
validates that the given path is the top directory of a Git repository.
Without this argument GitRepo also accepts subdirectories of a Git
repository (in which case GitRepo will point to the parent directory
that is the top directory of this repository) which may have undesired
in some cases.

(From OE-Core rev: 044c81bd916fbe7140d184eb103f74786cfef604)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-25 23:03:47 +01:00
Markus Lehtonen 618a2ede75 oeqa.utils.git: implement GitRepo.get_current_branch()
(From OE-Core rev: dcba2302adab47b398f1ce7d09c38828ea9ae426)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-25 23:03:46 +01:00
Markus Lehtonen 6cf74643e9 oeqa.utils.git: introduce GitRepo.rev_parse()
(From OE-Core rev: 55726e931536ed0cbd7b80588060b05a3145c934)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-25 23:03:46 +01:00
Markus Lehtonen 7fcc9f5ead oeqa.utils.git: support git commands with updated env
Extend GitRepo.run_cmd so that the caller may redefine and/or define
additional environment variables that will be used when the git command
is run.

(From OE-Core rev: 9b3c7c47f5d0fa473fe1db81b59b26531414781c)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-25 23:03:46 +01:00
Ross Burton 8670979d22 oeqa/runtime/rpm: use su instead of sudo
This test works fine with su, which is more likely to be installed in images
than sudo.

(From OE-Core rev: 59d10be745a1f7d31c68e4d5da9e1c3461b7d390)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-25 23:03:45 +01:00
Richard Purdie 3e97690117 parselogs: Ignore uvesafb timeouts
We're periodically seeing uvesafb timeouts on the autobuilder. Whitelist these
errors as there is little it seems we can do about them and we therefore
choose to ignore them rather than fail the builds.

[YOCTO #8245]

There is a better solution proposed in the bug with a -1 timeout however
this avoids failed builds until such times as that is implemented.

(From OE-Core rev: 8097f2da79b7862733494d2321e3dfdb0880804d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-25 22:57:36 +01:00
Richard Purdie 94646f4828 oeqa/buildiptables: Switch from netfilter.org to yoctoproject.org mirror
We've had some upstream mirror instability so use our own mirror for the
iptables sources to ensure this doesn't affect the test results.

(From OE-Core rev: 25f6af8895d5f5c6dcedde0a21285d63522769c8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-23 17:59:43 +01:00
Markus Lehtonen 9cabc18016 oeqa.buildperf: fix crash when creating globalres.log
Fix a bug that was introduced when converting to unittest framework.

(From OE-Core rev: 3bdb7b2e512b2f160360e95ed5b2be3871ec0b4b)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-20 16:06:02 +01:00
Richard Purdie 189371f839 devtool/recipetool/meta: Adapt to bitbake API changes for multi-configuration builds
Unfortunately to implenent multiconfig support in bitbake some APIs
had to change. This updates code in OE to match the changes in bitbake.
Its mostly periperhal changes around devtool/recipetool

[Will need a bitbake version requirement bump which I'll make when merging]

(From OE-Core rev: 041212fa37bb83acac5ce4ceb9b7b77ad172c5c3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18 10:07:23 +01:00
Richard Purdie 26b0657b2f parselogs: Ignore amb_nb warning messages under qemux86*
(From OE-Core rev: 857f4ca134e4575e71993b4fa255ebafec612d1e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18 09:27:54 +01:00
California Sullivan 56e53db4d9 parselogs.py: Add failed to setup card detect gpio error on x86
This error has occurred on the MinnowBoard Max and Turbot since its
inception. It supposedly indicates a non-working SD card reader, but
ours works fine. Whitelist the error.

(From OE-Core rev: d577028a1d756b70da056dee73df657cf8000baf)

Signed-off-by: California Sullivan <california.l.sullivan@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18 09:27:54 +01:00
California Sullivan a9b2c9d83a parselogs.py: Add dmi and ioremap errors to ignore list for core2
These errors have been occuring since the introduction of the 4.4
kernel with no apparent functionality loss. Whitelist for now.

(From OE-Core rev: 47b9058994f15507fc18ce0b08ac82a4c052966e)

Signed-off-by: California Sullivan <california.l.sullivan@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18 09:27:53 +01:00
California Sullivan e290a83882 parselogs.py: Ignore Skylake graphics firmware load errors on genericx86-64
These errors can't be fixed without adding the firmware to the initramfs
and building it into the kernel, which we don't want to do for
genericx86-64. Since graphics still work acceptably without the firmware
blobs, just ignore the errors for that MACHINE.

(From OE-Core rev: d73a26a71b2b16be06cd9a80a6ba42ffae8412c4)

Signed-off-by: California Sullivan <california.l.sullivan@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18 09:27:53 +01:00
Bill Randle f479e3866d testimage: allow using kvm when running qemux86* machines
Using kvm can provide significant speedups when running qemux86* machines
on an x86* host. Enabled by using the new QEMU_USE_KVM variable.

[YOCTO #9298]

(From OE-Core rev: ebac2c8d1fcd09ebce0659a4abb445e4f1c18571)

Signed-off-by: Bill Randle <william.c.randle@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18 09:27:53 +01:00
Markus Lehtonen 9800b4d9ff oeqa.buildperf: use oe.path.remove()
Drop the self-baked force_rm() method.

(From OE-Core rev: c86bf80abd87acb0da5860806822c64ec9dee089)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-17 10:35:44 +01:00
Markus Lehtonen 51970d10d6 oeqa.buildperf: be more verbose about failed commands
Log failures of commands whose output is stored.

(From OE-Core rev: 240f6e7366c8a9ea830e531d307dd2e27a61a6bd)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-17 10:35:44 +01:00
Markus Lehtonen d82a795683 oeqa.buildperf: introduce runCmd2()
Special runCmd() for build perf tests which doesn't raise an
AssertionError when the command fails. This causes command failures to
be detected as test errors instead of test failures. This way "failed"
state of tests is reserved for future making it possible to set e.g.
thresholds for certain measurement results.

(From OE-Core rev: 09590ac76a19ee1b1b4a9188f7fce5029f0de52a)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-17 10:35:44 +01:00
Markus Lehtonen 979be848e2 oeqa.buildperf: convert test cases to unittest
This commit converts the actual tests to be compatible with the new
Python unittest based framework.

(From OE-Core rev: 4e81967131863df7ee6c8356cb41be51f1b8c260)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-17 10:35:44 +01:00
Markus Lehtonen 09b9a4aeee oeqa.buildperf: add BuildPerfTestResult class
The new class is derived from unittest.TextTestResult class. It is
actually implemented by modifying the old BuildPerfTestRunner class
which, in turn, is replaced by a totally new simple implementation
derived from unittest.TestRunner.

(From OE-Core rev: 89eb37ef1ef8d5deb87fd55c9ea7b2cfa2681b07)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-17 10:35:44 +01:00
Markus Lehtonen 3acf648f58 oeqa.buildperf: add BuildPerfTestLoader class
(From OE-Core rev: b281c4a49b0df1de9b3137efb8ff50744e06c48d)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-17 10:35:43 +01:00
Markus Lehtonen 3f519df38e oeqa.buildperf: derive BuildPerfTestCase class from unitest.TestCase
Rename BuildPerfTest to BuildPerfTestCase and convert it to be derived
from TestCase class from the unittest framework of the Python standard
library. This doesn't work with our existing testcases or test runner
class and these need to be modified, too.

(From OE-Core rev: b0b434210a3dbd576f68344e29b8c20d18561099)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-17 10:35:43 +01:00
Markus Lehtonen daee7558a5 oeqa.buildperf: rename module containing basic tests
(From OE-Core rev: 56e455cf4b42ff4db36debd342bcb03c5199ba52)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-17 10:35:43 +01:00
Jose Perez Carranza 6203a77a53 buildperf: Add support for times without decimal part
Add logic for the cases when the time retrieved does
not have decimal part.

(From OE-Core rev: a6c9e515f8bc590612e3082ab1c4c254711c8e3b)

Signed-off-by: Jose Perez Carranza <jose.perez.carranza@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-17 10:35:43 +01:00
Ross Burton a191285b97 lib/oeqa/selftest/bbtests: don't report expected failures
Another instance where expected failures need to be not reported to the error
reporting service.

(From OE-Core rev: bb1cbb8d5bd7639554edcddf1d2eac4abdbb48c7)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-17 10:35:43 +01:00
Aníbal Limón dde033ab4c oeqa/runtime/syslog.py: Improve test_syslog_logger on systemd
When an image uses systemd journald acts as a main syslog daemon using
/dev/log.

The test_syslog_logger try to log a predifined message into the syslog
using logger and then search using grep in /var/log/messages if this
fails for some reason (file rotated) now search the predifined message
into the journal.

(From OE-Core rev: 26d7e5060a35d20df6f2586b70ed8d2853cc0186)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-17 10:35:43 +01:00
Richard Purdie 896f1c7696 oeqa/oetest: Improve subprocess error reporting
Without this, we get to know the command failed and the exit code but
have no idea how the command failed since we don't get the output by
default.

This makes it much easier to see what went wrong and stand a chance of
fixing it.

(From OE-Core rev: b020b01d41ccaae5d679f1f7950af2e1a1788d39)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-17 10:23:43 +01:00
Stephano Cetola 6b66e9317f Allow for simultaneous do_rootfs tasks with rpm
Give each rootfs its own RPM channel to use.  This puts the RPM metadata
in a private subdirectory of $WORKDIR, rather than living in DEPLOY_DIR
where other tasks may race with it.

This allows us to reduce the time that the rpm.lock is held to only the
time needed to hardlink the RPMs, allowing the majority of the rootfs
operation to run in parallel.

Also, this fixes the smart tests by generating an index for all packages
at the time of the test, rather than using the one provided by the
rootfs process.

Original credit for the enhancement should go to Steven Walter
stevenrwalter@gmail.com.

(From OE-Core rev: a92c196449c516fe51786d429078bbb1213bb029)

Signed-off-by: Stephano Cetola <stephano.cetola@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-12 15:25:22 +01:00
Aníbal Limón 3ac3f3f309 oeqa: start() add remaining args SimpleRemoteTarget and QemuTinyRunner
Now Runner's support extra_bootargs for the kernel so add
extra_bootparams to the start() methods to avoid exception.

(From OE-Core rev: 4c28c03a2322fbcb9a5c268b08eaeb71d940ee04)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-10 10:46:34 +01:00
Aníbal Limón feb125eb17 oeqa/utils/httpserver.py: HTTPServer enable thread connection handling
HTTPServer now supports multiple connections using Python threads.

(From OE-Core rev: 1d45b7bd611b900bc00530144ec0634307b1314f)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-10 10:46:32 +01:00
Ross Burton 9ec7cef3d0 oeqa/selftest: disable report-error class when builds are expected to fail
Some invocations of bitbake are expected to fail, so we don't want to report the
errors to errors.yoctoproject.org.  Also rewrite the messages in
test_invalid_patch so they reflect reality.

[ YOCTO #10052 ]

(From OE-Core rev: 51f74a0d1ce4de9d311becee8e7d7cc7cd703d45)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-10 10:46:31 +01:00
Mariano Lopez 37f1e0c154 oeqa/utils/sshcontrol.py: Allows to copy symlinks to target
Currently when copying a symlink to the target it will fail
throwing an exception. This will recreate symlinks from the
system performing the tests to the device under tests.

[YOCTO #9932]

(From OE-Core rev: 5705b7a55bc300e14c34b0530f4d49df101edd3c)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-04 15:22:23 +01:00
Mariano Lopez 65459f5b6d oeqa/oetest.py: Allow to export packages using symlinks
Currently packages that contains symlinks can't be extracted
and exported. This allows to export extracted such packages.

A nice side effect is improved readability.

[YOCTO #9932]

(From OE-Core rev: 0338f66c0d246c3b8d94ac68d60fbc4c314e500b)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-04 15:22:22 +01:00
Mariano Lopez 65a03c3f94 oetest.py: Remove check for __init__.py
When using "auto" in TEST_SUITES there is a check
for __init__.py in the <layer>/lib/oeqa/runtime/
directory in all the layers in BBLAYERS.

This check was needed because the way that python 2
import the modules, now that bitbake uses python 3
there is no need these __init__.py files, moreover
these files won't allow to import tests from other
layers.

This patch removes the check.

[YOCTO #9996]

(From OE-Core rev: f1cc272e4851fd994e9d052628a747ac19f90488)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-04 15:22:22 +01:00
Mariano Lopez 84d854c98b oetest.py: Add command line parameter support for tag in testexport
This allows to use a command line argument to change the
tag used to filter test instead of rebuilding the tests.

[YOCTO #8532]

(From OE-Core rev: 928e0eecdb126f7d0bacd05b7057fc825e0d8f05)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-04 15:22:22 +01:00
Mariano Lopez 44ae8284d0 selftest/runtime-test.py: Add test for testexport SDK feature
This adds test_testexport_sdk() to test the SDK feature
of testexport in the CI in order to avoid breaking it.

[YOCTO #9765]

(From OE-Core rev: badec3d10fcdd2d000450ab533caadcff1df5e13)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-04 15:22:22 +01:00
Aníbal Limón 1981ab0829 oeqa/utils/commands.py: Command class improve validations/decoding in output
When run a command sometimes the output isn't provided so validate
before trying to encode to utf-8, also some output like BIOS/EFI
contains characters that can't be codified into utf-8 for this reason
set errors='replace'.

[YOCTO #10019]

(From OE-Core rev: f2a04faf3c5d0a3cc562061b22e1c4873e1ca769)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-01 11:47:13 +01:00
Aníbal Limón 1648504198 oeqa/runtime/syslog: test_syslog_logger Don't try to use logread when systemd is enabled
Busybox logread uses shmmem circular buffer to retrive [1] syslog messages
when systemd is enabled this shmem circular buffer isn't enabled because
systemd journald doesn't provide it.

[1] https://git.busybox.net/busybox/tree/sysklogd/logread.c?id=accd9eeb719916da974584b33b1aeced5f3bb346#n121

(From OE-Core rev: f49e4847ba00cdd072e5f072cb9ca69ef98af758)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-01 11:47:13 +01:00
Aníbal Limón b4ba36a9d7 oeqa/runtime/syslog.py: Improve test_syslog_logger
Instead of make all the testing in a shell one liner, divide the
test into 3 operations to be able to know in what part is failing.

Parts,
	- Log message to syslog
	- Review if message exist in /var/log/messages
		- Review if message exist using logread

(From OE-Core rev: f3fe3590e887ee311f23723103eca41dcf58aa8b)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-01 11:47:13 +01:00
Mariano Lopez f31683d5ba selftest/runtime-test.py: Add test for import test from other layers
There are some features in testimage/testexport that are not tested;
this might lead to break some of these features without notice.

This adds a new test in order to test two features of testimage:
  - Import test from other layers.
  - Install/Unistall in the DUT without a package manager.

[YOCTO #9764]
[YOCTO #9766]

(From OE-Core rev: cffab2257dacfa741c64611dfdf361f77aff9460)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-01 11:47:12 +01:00
Paul Eggleton cc3c276852 oe-selftest: recipetool: add tests for git URL mangling
Add three tests to verify that the git URL mangling is working the way
it's supposed to. This should prevent us regressing on this again in
future.

(From OE-Core rev: d8d01f462ddbb79cff23b544fcd0ce251f05f8ce)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-01 11:47:10 +01:00
Ross Burton 1b4dc787b8 oeqa/selftest/signing: check that we have GPG in setup
(From OE-Core rev: 6eb6fde2567a2692afccb7ee0546b2c992c168aa)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:56:31 +01:00
Ross Burton 5218c24c8b oeqa/selftest/signing: use a temporary directory for GPG home
Instead of using a directory in the layer as the GPG home and carefully deleting
the right files from it, use tempfile to create a temporary directory which will
be cleaned up for us.

Also change the public/secret key variables to be absolute paths as they're
always used as absolute paths.

(From OE-Core rev: d4a5b5d11c6d7d5aba5f2eb88db091c1b98ef87c)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:56:31 +01:00
Ross Burton 31cef0d518 lib/oeqa/decorators: handle broken links when creating new symlink
When checking if a link exists before creating it, use os.path.lexists() as
otherwise os.path.exists() on a broken link will return False.

(From OE-Core rev: ec24b6de2b8686e1f779fef3a963e66f70eeba74)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:56:25 +01:00
Jose Perez Carranza 8da54b401d oeqa/controllers: Add test target for Systemd-boot
A new test target is neede to test Systemd-boot similar
to gummiboot. Created a copy of GummibootTarget class
and named as SystemdbootTarget, at this point the
gummibootTarget will remain until documentation is
updated with new systed information.

(From OE-Core rev: d1fd3a7dbc363bdc02d810f796b3b810cb13f029)

Signed-off-by: Jose Perez Carranza <jose.perez.carranza@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-21 07:47:52 +01:00
Mariano Lopez cd78d8745d oetest.py: Fix problem when there is more than one json file for package extraction
This allow to have more than one test to have json file in order
to install packages in the DUT without using a package manager.

[YOCTO #9926]

(From OE-Core rev: 129c2b4589710aa24e23108f2728f1625b72cb06)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-21 07:47:52 +01:00
Mariano Lopez e1aa836bb9 lib/oeqa/utils/commands.py: Move updateEnv() from runexported.py
updateEnv() can be used in other places so move the
function to utils/commands.py

(From OE-Core rev: 380e5d80898cac4ffc9715b3f597d0b62a0643ff)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-21 07:47:52 +01:00
Ross Burton bb42984277 oeqa/recipetool: update recipe test to pass SHA
(From OE-Core rev: 71dd4c05c41e8b363dc1ecac1f5105d316ee82dc)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-20 10:28:53 +01:00
Ross Burton 738f47a2d9 oeqa/devtool: update recipe test as libmatchbox changed
(From OE-Core rev: b36712eef14c20007e0adb01cc7d4bce9e7926bb)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-20 10:28:53 +01:00
Jussi Kukkonen 096b4c639b libxsettings-client: Remove as unneeded
xsettings-client is not meant to be a shared library and the only
user in oe-core (libmatchbox) now has an in-tree copy.

(From OE-Core rev: ed5cbf977dca8f3d85c8d933071c6e49fb4bd95f)

Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-20 10:28:50 +01:00
Ed Bartosh 05084640e6 wic: oe-selftest: build bmap-tools-native
bmap-tools-native is required to generate .bmap file. Without it
wic fails with this error when run with --bmap option:
    Error: A native program bmaptool required to build
           the image was not found

Added bmap-tools-native to the list of requirements to build
in Wic.setUpLocal method.

(From OE-Core rev: dd11ab551f9bf1d8761d0430f19510a9caa53a12)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-20 10:28:50 +01:00
Paul Eggleton d62fd7711d devtool: add finish subcommand
Add a subcommand which will "finish" the work on a recipe. This is
effectively the same as update-recipe followed by reset, except that the
destination layer is required and it will do the right thing depending
on the situation - if the recipe file itself is in the workspace (e.g.
as a result of devtool add), the recipe file and any associated files
will be moved to the destination layer; or if the destination layer is
the one containing the original recipe, the recipe will be overwritten;
otherwise a bbappend will be created to apply the changes. In all cases
the layer path can be loosely specified - it could be a layer name, or
a partial path into a recipe. In the case of upgrades, devtool finish
will also take care of deleting the old recipe.

This avoids the user having to figure out the correct actions when
they're done - they just do "devtool finish recipename layername" and
it saves their work and then removes the recipe from the workspace.

Addresses [YOCTO #8594].

(From OE-Core rev: fa550fcb9333d59b28fc0e4aebde888831410f5c)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-20 10:28:49 +01:00
Aníbal Limón ad6aaae4bf utils/qemurunner.py: QemuRunner.start() add support for specify extra kernel cmdline
Add ability to specify extra_bootargs (kernel cmdline) in order to enable systemd
debug log in images that enables systemd init.

[YOCTO #9299]

(From OE-Core rev: 09d62551c289b5607341a4f9c46eecd6390ad774)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-20 10:28:47 +01:00
Ross Burton 38d2c59ab8 remove sukoku
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-20 10:28:44 +01:00
Ross Burton 9c4b143607 oeqa/sdk: remove buildsudoku test
This application needs GTK+ 2 which we're trying to move away from for Sato
images.

(From OE-Core rev: 5e52f208a17e7122209675fa96de6e7807c62feb)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-20 10:28:44 +01:00
Maxin B. John 78523811fb buildoptions.py: remove directfb image build test
removing this test since we move directfb out of oe-core

(From OE-Core rev: 2d8fda36ecfa1945f22b7139a2febd12ec59272b)

Signed-off-by: Maxin B. John <maxin.john@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-10 14:12:17 +01:00
Paul Eggleton a14d30f7de oe-selftest: drop test_prepare_unexpected
This test refers to a function that no longer exists after the eSDK
install double execution of bitbake has been removed, and since
test_prepare_unexpected is the only test in this module, drop the
entire module. We can easily resurrect it if we have unit tests to add
in the future.

(From OE-Core rev: 7e792a22e62904ed2dafb1ea214911235e3f3efc)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 10:29:44 +01:00
Richard Purdie 1bbd580a37 oeqa/selftest/bbtests: Fix failing test after progress changes
The progress patches change the output slightly, update the test to
deal with this.

(From OE-Core rev: 90dbd838fa97c89ace5cb147aa5cff39b94178b1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:26 +01:00
Ross Burton c7147ceec8 oeqa/runtime/systemd: fix typo in skipUnlessPassed decorator
(From OE-Core rev: f0420d8c934154a94957cabcfdc993f0eae92cd7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:26 +01:00
Ross Burton 5c11e365e1 lib/oeqa: add Galculator to SDK and runtime tests
This is a GTK+3 application, so we don't need to ship GTK+ 2 in Sato just for
the SDK test suite.

[ YOCTO #9780 ]

(From OE-Core rev: 53fcfe4348a2ca727844f2b0bd3fca2902cbdda0)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-01 16:22:48 +01:00
Ross Burton b305564028 oeqa/runtime/syslog: remove redundant skipUnlessPassed
The test test_syslog_help doesn't exist, so skipping unless it passed just
produced a warning.

(From OE-Core rev: 9856fe02bf30cd9e834a7d324e357070f4d7735f)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-01 16:22:48 +01:00
Ross Burton 076a93eed7 oeqa/utils/commands: fix single-character variable matching in get_bb_vars()
(From OE-Core rev: 2f7e4d0636ce1647edf2d82c65b82be0eb2c1db1)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-01 16:22:48 +01:00
Markus Lehtonen eb36f00002 oe-build-perf-test: implement --globalres-file option
Using this option the script appends test results into a 'global results
file'. A CSV-formatted output of the results. This option is to provide
compatibility with the old build-perf-test.sh.

(From OE-Core rev: e9f18e63220e452f2b0c878998e57d944ae83980)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-01 16:22:47 +01:00
Markus Lehtonen 8c596369dd oeqa.buildperf: add git revision and branch to result data
BuildPerfTestRunner determines these from the Git repository under which
it is being run (i.e. where the build directory exists). The branch and
revision may be defined/overridden with OE_BUILDPERFTEST_GIT_BRANCH
and OE_BUILDPERFTEST_GIT_BRANCH environment variables, if needed. This
makes it possible to run the build performance test script even if the
top directory is not a git repository clone, for example.

(From OE-Core rev: e6004582454d8c6a18f617c12e6e408ded5be8df)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-01 16:22:47 +01:00
Markus Lehtonen 964fffa514 oeqa.utils: add git module
A new helper module for easier interaction with Git repositories.
Provides GitRepo class that represents one local Git repository clone.
The GitRepo class currently only has one method, run_cmd(), for running
arbitrary git commands in the repository. More specialized methods for
commonly used git operations can be added later.

(From OE-Core rev: 4eaf434f885afbda03fe67ab6e9ff291c7a9c77e)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-01 16:22:47 +01:00
Markus Lehtonen 5599645d33 oeqa.buildperf: archive build/conf into test results
Have the build/conf directory as part of test results.

(From OE-Core rev: ea32a8a65f668450e0ec0e80483bd5e8c91b8a18)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-01 16:22:47 +01:00
Markus Lehtonen da3b9242fc oeqa.buildperf: add test Test4
Re-implement "test4" from build-perf-test.sh which measures eSDK metrics.

(From OE-Core rev: c449ca1b062426380a0572aed2da4ee4a83281f5)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-01 16:22:47 +01:00
Markus Lehtonen fe24b5a597 oeqa.buildperf: add test Test3
Re-implement "test3" from build-perf-test.sh which measures
bitbake parsing time.

(From OE-Core rev: 388c7160b90247df9c78d8803083821402c13af5)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-01 16:22:47 +01:00
Markus Lehtonen 0b2fae81fd oeqa.buildperf: add test Test2
Re-implement "test2" from build-perf-test.sh which measures
'bitbake core-image-sato -c rootfs'.

(From OE-Core rev: 8cb322d6a5be08f0bb2c9e5d3b73161b20d737a3)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-01 16:22:47 +01:00
Markus Lehtonen 1701a34f14 oeqa.buildperf: add test Test1P3
Re-implement "test1_p3" from build-perf-test.sh which measures
'bitbake core-image-sato' with rm_work enabled.

(From OE-Core rev: bfe48d343e8a281760da0b6744c08d47039f93e0)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-01 16:22:47 +01:00
Markus Lehtonen b5ca437714 oeqa.buildperf: add test Test1P2
Re-implement "test1_p2" from build-perf-test.sh which measures
'bitbake virtual/kernel'.

(From OE-Core rev: 5ff82c6e2b0ff41a9bfdd49c99e84a982cf6e467)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-01 16:22:47 +01:00
Markus Lehtonen 2c1b87482c oeqa.buildperf: add test Test1P1
Re-implement "test1_p1" from build-perf-test.sh which measures
'bitbake core-image-sato'.

(From OE-Core rev: 1ddd4a8ce4e2053def0e946d4e4b4b97bdc28e61)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-01 16:22:47 +01:00
Markus Lehtonen 1d88659ef6 oeqa.buildperf: implement BuildPerfTestRunner class
The new class is responsible for actually running the tests and
processing their results. This commit also adds a decorator function for
adding new tests. No automatic test discovery, at least yet.

(From OE-Core rev: bf90aecb7e150d6bfac7240286c797b79d26528b)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-01 16:22:46 +01:00
Markus Lehtonen 6512d6956b oeqa.buildperf: add method for saving buildstats
(From OE-Core rev: 15b44a484ce408b0a1acfb907e2b12ebc6cce296)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-01 16:22:46 +01:00
Markus Lehtonen 45c6a04a37 oeqa.buildperf: add method for measuring file disk usage
Add a new method to BuildPerfTest class for measuring the disk usage of
a file of directory.

(From OE-Core rev: 85cdc240e75d481e93238fbf75f8b8431da05f19)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-01 16:22:46 +01:00
Markus Lehtonen 1a0e20546e oeqa.buildperf: add method to log shell commands
Add new methods to BuildPerfTest class for running a shell
command and logging its output.

(From OE-Core rev: 8f0b11ba1266f9c650cf34d9b394d72009ee7207)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-01 16:22:46 +01:00
Markus Lehtonen 6e27b2ae0e oeqa.buildperf: method for measuring system resource usage
Extend BuildPerfTest class with a new method for measuring the system
resource usage of a shell command to BuildPerfTest class.  For now,
easurement of the elapsed time is done with the Gnu time utility,
similarly to the build-perf-test.sh shell script. And, it currently only
records the elapsed (wall clock).

The measured values (currently, only the elapsed time) is actually a
dictionary, making it possible to extend it with additional resource
values, e.g. cpu time or i/o usage, in the future.  In addition to the
actual values of the measurement each record contains a 'name' and
'legend' where name is supposed to function as a common key or id over
test runs, making comparison and trending easier, for example. Legend is
supposed to be a short human readable description.

(From OE-Core rev: ced156bfea4a6649d201f41275641a633f218322)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-01 16:22:46 +01:00
Markus Lehtonen 1b10ded015 oeqa.buildperf: add BuildPerfTest class
The new class will be used as an abstract base class for build
performance tests. This implementation contains some common
functionality used in multiple tests, "copied" from the
build-perf-test.sh shell script.

(From OE-Core rev: 35cd7363759a286e80ddca0d028db3d2bf524b17)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-01 16:22:46 +01:00
Markus Lehtonen cee685ca0d oeqa.buildperf: functionality to drop kernel caches
Add a new utility class for dropping Linux kernel caches.  It uses sudo
and tee to write to the drop_caches file. Checking if the user has the
permissions to drop caches (without a password) is done by trying to
writing an invalid value to the drop_caches file. This way, we will find
if writing (with tee) is possible but not really dropping caches, yet.

User can avoid giving the password by adding something like:
<user> ALL = NOPASSWD: /usr/bin/tee /proc/sys/vm/drop_caches
to the system sudoers file.

(From OE-Core rev: c9cb248429ced50c96d11ba5361c272d4c9b9323)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-01 16:22:46 +01:00
Markus Lehtonen 7d42ceae3a oe-build-perf-test: introduce oeqa.buildperf module
Wireframe of a new Python module for containing build performance tests
and utility functions.

(From OE-Core rev: b44af9051a9da8d2cddc7b071a896331967f15cc)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-01 16:22:46 +01:00
Markus Lehtonen ec8be2039e oeqa.utils.commands: runCmd: return stderr output, too
Useful if one wants to separate stdout and stderr.

(From OE-Core rev: de9744c91a997a5ab0e7a19dbe13d8def8d62800)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-01 16:22:46 +01:00
Markus Lehtonen cf4c8980d7 oeqa.utils.commands: use get_bb_vars() in get_bb_var()
Get rid of duplicate code.

(From OE-Core rev: cdd6b7386afd460337705d8117a4328d4993ecef)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-01 16:22:45 +01:00
Markus Lehtonen c9639e6523 oeqa.utils.commands: Introduce get_bb_vars()
A new function for getting values of multiple bitbake variables at the
same time.

(From OE-Core rev: fe3039322e2f846b336ac5af5177e9da27d79695)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-01 16:22:45 +01:00
Ross Burton 1960abc96c oeqa/targetbuild: use os.path.join instead of string concatenation
(From OE-Core rev: 1d40f6e3d85d0cd3be6dd784677686ed4dec89c4)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-23 14:26:16 +01:00
Ross Burton 40268fe27a oeqa/targetbuild: add extra_cmds argument to run_configure
Some upstreams need more than just gnu-configize ran before ./configure works,
such as ./autogen.sh or autoreconf.  Add extra_args (defaulting to
gnu-configize) so that this can be done in test cases.

(From OE-Core rev: 7096f2889f1623ce97a6696f6f4c7217f0efb972)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-23 14:26:16 +01:00
Ross Burton 0aa028f5a9 oeqa/targetbuild: fix folder name estimate logic
The re.sub() used to transform a tarball into a best guess folder name wasn't
right, as there isn't enough escaping and tar.xz was missing.

(From OE-Core rev: ac99135b009a1066486ed2afb2f298d0a5a3a854)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-23 14:26:16 +01:00
Ross Burton 675843d540 oeqa: fix hasPackage, add hasPackageMatch
hasPackage() was looking for the string provided as an RE substring in the
manifest, which resulted in a large number of false positives (i.e. libgtkfoo
would match "gtk+").

Rewrite the manifest loader to parse the files into a proper data structure,
change hasPackage to do full string matches, and add hasPackageMatch which does
RE substring matches.

(From OE-Core rev: b9409863af71899e02275439949e3f4cdfaf2d0f)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-23 14:26:14 +01:00
Mariano Lopez 4a842cc952 selftest/testexport.py: Add test test_testexport_basic
Exported tests is not been tested right now and it get
broken very easily with import errors from bitbake.

This adds a new test in order to verify in the CI if
the testexport gets broken.

[YOCTO #8384]

(From OE-Core rev: b6e0a762f8a17ee323cef02934f9a624faf706ed)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-07 21:36:20 +01:00