Commit Graph

1497 Commits

Author SHA1 Message Date
Jair Gonzalez f8745bb519 selftest/wic: reorganizing test methods by functionality
Part of the test methods were rearranged to group them by
functionality and identify more easily opportunities to extend
coverage.

(From OE-Core rev: f290a2c9b946e00dbc451592691596c656d49042)

Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-22 08:50:17 +00:00
Jair Gonzalez 285c2ed90e selftest/wic: code cleanup
The following changes were added to improve the code readability:

- Removed redundant backlashes between brackets
- Aligned continuation lines according to PEP8 style
- Refactored command execution in the next methods for enhanced
  legibility:

  - test_build_image_name(self)
  - test_gpt_image(self)
  - test_qemux86_directdisk(self)
  - test_rootfs_indirect_recipes(self)
  - test_iso_image(self)
  - test_mkgummidisk(self)
  - test_mkefidisk(self)
  - test_directdisk_bootloader_config(self)
  - test_qemu(self)
  - test_bmap(self)
  - test_systemd_bootdisk(self)
  - test_sdimage_bootpart(self)

(From OE-Core rev: 5ba21836fe3c061a6271f855a9e26c66004d330a)

Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-22 08:50:17 +00:00
Jair Gonzalez c48c0f38b1 selftest/wic: adding Testopia ID numbers to test cases missing it
The following test cases were assigned an ID number on Testopia:

1496 Test generation of .bmap file
1560 Test creation of systemd-bootdisk image
1561 Test creation of sdimage-bootpart image

(From OE-Core rev: b53e432206eaba7c6c67e4689c25b5e62d7ee9b2)

Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-22 08:50:17 +00:00
Maciej Borzecki 5903182484 oeqa/utils/commands.py: allow use of binaries from native sysroot
Tests may need to run a native tool that is not available on the host
filesystem, but can be built using one of the *-native recipes. In such case,
the tool will be available in native sysroot, and running in from that location
will require adjustments to PATH.

runCmd() can now take a path to native sysroot as one of its arguments and
setup PATH accordingly.

(From OE-Core rev: f2a04631949db72d4261d1c142c5044fad3741f9)

Signed-off-by: Maciej Borzecki <maciej.borzecki@rndity.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-22 08:50:16 +00:00
Richard Purdie 8ccf396c72 Revert "selftest/wic: extending test coverage for WIC script options"
This reverts commit 68cb3180c1b0dcee50812b21f98850d188d8621b as this wasn't
ready for merge and there are new better versions.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-20 17:06:38 +00:00
Ed Bartosh 2f4008d694 oe-selftest: import git module only when needed
git module is not included into standard Python
library and therefore causes import errors on the systems
where PythonGit is not installed.

As git module only used in the code implementing --repository
functionality it's better to import git only in the scope
that requires it.

[YOCTO #10821]

(From OE-Core rev: 66be32c1a075201d6ee0e9b9e10b84e6a2ace745)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-20 15:22:17 +00:00
Jair Gonzalez aa1c50a32b selftest/wic: extending test coverage for WIC script options
The previous WIC script selftest didn't cover all of its command
line options. The following test cases were added to complete
covering them:

1552 Test wic --version
1553 Test wic help create
1554 Test wic help list
1555 Test wic list images
1556 Test wic list source-plugins
1557 Test wic listed images help
1558 Test wic debug, skip-build-check and build_rootfs
1559 Test image vars directory selection
1562 Test alternate output directory

In addition, the following test cases were assigned an ID number on
Testopia:

1560 Test creation of systemd-bootdisk image
1561 Test creation of sdimage-bootpart image

Finally, part of the test methods were rearranged to group them by
functionality, and some cleanup was made to improve the code's
compliance with PEP8 style guide.

Fixes [YOCTO 10594]

(From OE-Core rev: 3d2ac67765020885a0996ebdd97a576ba37dbec0)

(From OE-Core rev: 68cb3180c1b0dcee50812b21f98850d188d8621b)

Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-20 15:22:16 +00:00
Ross Burton de43650c29 lib/oe/package_manager: bail if createrepo can't be found
If createrepo isn't found then the errors later are mysterious, so explicitly
check and error out early if it isn't there.

(From OE-Core rev: e09636bbb3ea8ec58984197fd9c691bb908efe00)

(From OE-Core rev: c87361fc886432a9db584712bf3e41ecd0541960)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-17 09:57:04 +00:00
Ross Burton 1969854ec3 utils: Always use datastore's PATH for host_gcc_version
BUILD_CC may reference something like ccache and expect this to come from
ccache-native, we at least have some selftests which assume this. Modify the
code to use PATH when runnig BUILD_CC to ensure the tests continue to work
as expected.

(From OE-Core rev: f3e753372baac43d0921186340cf260df056de20)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:23 +00:00
Joshua Lock 3c59b1bf93 meta: remove True option to getVarFlag calls
getVarFlag() now defaults to expanding by default, thus remove the
True option from getVarFlag() calls with a regex search and
replace.

Search made with the following regex:
getVarFlag ?\(( ?[^,()]*, ?[^,()]*), True\)

(From OE-Core rev: 2dea9e490a98377010b3d4118d054814c317a735)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:23 +00:00
Joshua Lock c4e2c59088 meta: remove True option to getVar calls
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.

Search made with the following regex: getVar ?\(( ?[^,()]*), True\)

(From OE-Core rev: 7c552996597faaee2fbee185b250c0ee30ea3b5f)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:23 +00:00
Ed Bartosh 46e787b248 buildhistory-diff: report directory renames
The script detects directory renaming if two different
directories with the same set of files are added and removed.

[YOCTO #10691]

(From OE-Core rev: 944db779a9f45cbeeebc976c00da37a517eea237)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:23 +00:00
Ed Bartosh c21cec8488 uninative: rebuild uninative for gcc 4.8 and 4.9
Some c++ libraries fail to build if uninative is built
with gcc 5.x and host gcc version is either 4.8 or 4.9.

The issue should be solved by making separate uninative sstate
directory structure sstate-cache/universal-<gcc version> for host gcc
versions 4.8 and 4.9. This causes rebuilds of uninative if host gcc
is either 4.8 or 4.9 and it doesn't match gcc version used to build
uninative.

[YOCTO #10441]

(From OE-Core rev: d36f41e5658bbbb6080ee833027879c119edf3e0)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:23 +00:00
Jose Perez Carranza 333890953d postinst: Add a test case to verify postinst scripts behavior
Add test case that verify behavior of postinst scripts at
roofts time and when is delayed to the first boot directly
on the target.

(From OE-Core rev: 82b171f3b37e6733997fc1e7685b7cac5a3476e7)

Signed-off-by: Jose Perez Carranza <jose.perez.carranza@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:23 +00:00
Francisco Pedraza 23d1c4ffb7 selftest: Test needed to verify postinst order
It verifies the following:

1. Compile a minimal image.
2. The compiled image will add the layer with the recipe postinst,
previously created at:
"meta-selftest/recipes-test"
3. Run QEMU.
4. Validate the task execution order.
[YOCTO #5319]

(From OE-Core rev: a8ff789a3bfedcbc4358db7907a45270d8b1b76a)

Signed-off-by: Francisco Pedraza <francisco.j.pedraza.gonzalez@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:23 +00:00
Ross Burton d535aba974 oeqa: move lib/oe tests to oe-selftest
These tests don't get ran often (as demonstrated by the fact that some were not
ported to Python 3), so move them to oeqa/selftest so they get executed
frequently and can be extended easily.

[ YOCTO #7376 ]

(From OE-Core rev: 2001979ad41e6fdd5a37b0f90a96708f39c9df07)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:22 +00:00
Leonardo Sandoval 030f62e9c8 selftest: buildoptions: skip read-only-image test depending on distro
Poky-tiny cannot build core-image-sato, so skip test (read-only-image)
in this case.

(From OE-Core rev: bcee8c614f28b38054f5d8c1c5251b3702cf113c)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:22 +00:00
Leonardo Sandoval e2ea2597c9 selftest: bblayers: remove linux kernel checks for show-recipes check
Preferred kernel recipes depends on the distro, so remove the kernel
checks to avoid failures on non-poky distros and make the test
distro agnostic.

(From OE-Core rev: ae92b72990b3ac804791b501d08126491fdddb7c)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:22 +00:00
Leonardo Sandoval 279c8ce310 selftest: bbtests: use minimal image so all distros can execute it
poky-tiny distro cannot build full-cmdline image, so use an image
(core-image-minimal) that can be built in all distros.

(From OE-Core rev: b293dd4200bbb3705c88af6113be7f43fbd0ed72)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:22 +00:00
Leonardo Sandoval 009adf73fa selftest: base: new object member to store the DISTRO value
Instead of quering it multiple times, query once and use it on
test method skip checks. Also, rename current distro sstate object
member to a more meaninful name.

(From OE-Core rev: bde9d99575a63ad2d7fd5974ce6ce19aad9a8984)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:22 +00:00
Mariano Lopez a92d27acd8 oeqa/utils/commands.py: Fix get_bb_vars() when called without arguments
Commit 9d55e9d489 broke calling get_bb_vars()
when called without arguments. This fix this issue.

(From OE-Core rev: 91f856426c7523e1ebdf6d6f93f5fa7e509d6e49)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 16:15:04 +00:00
Paul Eggleton 66adb6bdf7 oe-selftest: add basic tinfoil tests
Add some tests to verify that the new tinfoil API is operating
correctly.

(From OE-Core rev: 16afda66b861ba028c1152dcdcab2b7ebfbff965)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:30:50 +00:00
Paul Eggleton 0cba3cbcb4 oe-selftest: devtool: improve test_devtool_modify slightly
* Check that man .in file actually gets modified, since sed -i doesn't
  fail if it it doesn't
* Use a variable for man file path

(From OE-Core rev: 9ad36e945fa5b03726f78ba99e823eade8daa710)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:30:50 +00:00
Paul Eggleton 078ef4361d lib/oe/recipeutils: drop parse_recipe_simple()
This was intended to be used with tinfoil, but tinfoil now has its own
parse_recipe() method to do this which works properly in the memres
case.

(From OE-Core rev: cdfc6173cb06ca374b7d927442a0fdde8373ba48)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:30:50 +00:00
Paul Eggleton 275b3fe5f4 classes/patch: move several functions to oe.patch
Move patch_path(), src_patches() and should_apply() to oe.patch, making
them easier to call from elsewhere (particularly across the
UI/server boundary).

(From OE-Core rev: 2724511e18810cc8082c1b028e3b7c8a8b5def56)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:30:50 +00:00
Paul Eggleton 1f8cf63f8a oe-selftest: use tinfoil.parse_recipe()
Use tinfoil.parse_recipe() in order to allow oe-selftest to be used in
memres mode.

(From OE-Core rev: 499ee9006271112f22cfe08fa5ba5c21be95380b)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:30:49 +00:00
Paul Eggleton 93d675304d oe-selftest: make tinfoil quiet when using to start QEMU
We don't need to see the parsing/cache loading message in the
oe-selftest output, so use the newly added quiet option to disable it.

(From OE-Core rev: c023bc55ff000d1de891d1a8e2a163e94bf63de6)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:30:49 +00:00
Paul Eggleton 557382ab42 lib/oe/recipeutils: use cooker function instead of bb.providers
We now have a function in cooker itself that can do this lookup;
additionally, the rewritten tinfoil's cooker adapter has its own
implementation that can work remotely, so if we use it then this
function can work in that scenario as well.

(From OE-Core rev: 0a6a4be99c1e4ef3c0da53d63f18ad579545d6a8)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:30:49 +00:00
Paul Eggleton af867199a5 classes/image: suppress log_check mechanism for warnings/errors logged through BitBake
If you printed a warning through bb.warn() / bbwarn or an error through
bb.error() / bberror, this was also being picked up by our log_check
mechanism that was designed to pick up warnings and errors printed by
other programs used during do_rootfs. This meant you saw not only the
warning or error itself, you saw it a second time through log_check,
which is a bit ugly. Use the just-added BB_TASK_LOGGER to access the
logger and add a handler that we can use to find out if any warning or
error we find in the logs is one we should ignore as it has already been
printed.

Fixes [YOCTO #8223].

(From OE-Core rev: fb37304d27857df3c53c0867e81fbc8899b48089)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:30:49 +00:00
Paul Eggleton 13916a4fab lib/oe/rootfs: fix log_check warnings being printed twice with RPM packaging
We were calling _log_check() in the RPM-specific rootfs class as well as
in the base class; this is unnecessary and resulted in any errors/warnings
generated during the actual package installation time triggering two warnings
instead of one. Drop the call from RpmRootfs._create() to fix this.

(From OE-Core rev: 541c56d755ba0354297673e857628026ad9e4df2)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:30:49 +00:00
Ola x Nilsson d65886bb69 oe-pkgdata-util: Make read-value handle override variables
Some variables in pkgdata files have a package-name override.  When
the bare variable can not be found, try with the override-variant.

PKGSIZE is one such variable, and already had special code to handle this.

Test included.

(From OE-Core rev: 6df99cda894033cba68bc6ab91e47f67e0d788a5)

Signed-off-by: Ola x Nilsson <ola.x.nilsson@axis.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:30:49 +00:00
California Sullivan 840ea755f2 parselogs.py: Don't clog QA with Joule errors
The Joule is very new hardware and there is ongoing kernel and firmware
work to fix these issues, which will be available in future kernel and
firmware releases. In the meantime, don't clog QA reports.

[YOCTO #10611]

(From OE-Core rev: facf9fa905100945738c13f9f79e938ed4a81030)

Signed-off-by: California Sullivan <california.l.sullivan@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-13 22:55:21 +00:00
Patrick Ohly 9eb89487ff buildstats.py: skip collecting unavailable /proc data
Some virtualized environments like Linux-VServer do not have the
entries under /proc that the new system usage sampling expected,
leading to an exception when trying to open the files.

Now the presence of these files is checked once before enabling the
corresponding data collection. When a file is missing, the
corresponding log file is not written either and pybootchart will not
draw the chart that normally displays the data.

Errors while reading or writing of data samples is intentionally still
a fatal error, because that points towards a bigger problem that
should not be ignored.

Reported-by: Andreas Oberritter <obi@opendreambox.org>
(From OE-Core rev: daeee2d6731014c33f0d1f8a3846830c099932b4)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-13 22:55:20 +00:00
Ola x Nilsson 2765231499 recipetool: selftest: Add test for recipetool plugin loading
Test that recipetool plugins are loaded in a well defined order.

(From OE-Core rev: 044de8424a454a7057906e44eb56e2134ebb17e4)

Signed-off-by: Ola x Nilsson <ola.x.nilsson@axis.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-13 22:55:20 +00:00
Ola x Nilsson 9170a88cbd devtool: selftest: add test for devtool plugin loading
Test that devtool plugins are loaded in a well defined order.

(From OE-Core rev: 0de81f0c8b29d8b442b3d099c3bec3fd345b6bfe)

Signed-off-by: Ola x Nilsson <ola.x.nilsson@axis.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-13 22:55:20 +00:00
Mariano Lopez 904b915369 oeqa/utils/metadata.py: Add metadata library
Adds functions to get metadata from the host running the tests.

[YOCTO #9954]

(From OE-Core rev: 10b05794254886e55c76f29f7778d783c550befa)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-13 22:55:19 +00:00
Mariano Lopez 9d55e9d489 oeqa/utils/commands.py: Make a copy of variables in get_bb_vars
The function get_bb_vars will remove items for the list passed
as the function argument, this will leave the caller with an
empty list and the function never says it will consume the items.

This hasn't been found before because only get_bb_var uses this
function.

(From OE-Core rev: 22b7fa24fefcc3974806d1b282c93b8c5880f6a4)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-13 22:55:19 +00:00
Juro Bystricky 90404e1f24 targetloader.py: drop test for ClassType
ClassType was removed from python3.
The code testing for ClassType kept throwing AttributeError exceptions:

    module 'types' has no attribute 'ClassType'

The exceptions prevented loading of any dynamically resolved target
controllers.

(From OE-Core rev: d62f18c39bc0ed3b0f5ac8465b393c15f2143ecf)

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12 15:16:42 +00:00
Stephano Cetola d8fbaebc48 package_manager: remove strings and migrate to direct arrays
When using subprocess call and check_output, it is better to use arrays
rather than strings when possible to avoid whitespace and quoting
problems.

[ YOCTO #9342 ]

(From OE-Core rev: b12cec9a5ef14ecb02be7feec65508cf5d65c795)

Signed-off-by: Stephano Cetola <stephano.cetola@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-08 10:31:30 +00:00
Ed Bartosh 1b32c6ed02 selftest: wic: fix test_qemu
Setting WKS_FILE variable in qemux86-64 made wic test to
use wrong wks file to produce an image and resulted in
test_qemu failure.

Used conditional assignment in qemux86-64 and explicitly
set WKS_FILE in wic testing suite to make the suite to use
wic-image-minimal.wsk. This should fix test_qemu failure.

(From OE-Core rev: 3bca4d18c2712e3b154bacfb917f0a749ebaddeb)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:38:06 +00:00
Patrick Ohly f1a527c5f6 buildstats: reduce amount of data stored for system utilization
Pre-processing /proc data during the build considerably reduces the
amount of data written to disk: 176KB instead of 4.7MB for a 20
minuted build. Parsing also becomes faster.

The disk monitor log added another 16KB in that example build. The
overall buildstat was 20MB, so the overhead for monitoring system
utilization is small enough that it can be enabled by default.

(From OE-Core rev: b17812385cd55e81066d3ceda92dffdc6e5564da)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:37:59 +00:00
Patrick Ohly 25a5536a6e buildstats: record disk space usage
Hooks into the new monitordisk.py event and records the used space for
each volume. That is probably the only relevant value when it comes to
visualizing the build and recording more would only increase disk
usage.

(From OE-Core rev: 21a5b569370f47cc02291e1d8b76fe43faa04ea6)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:37:59 +00:00
Patrick Ohly 8f475b78c9 buildstats: add system state sampling
/proc/[diskstats|meminfo|stat] get sampled and written to the same
proc_<filename>.log files as during normal bootchat logging. This will
allow rendering the CPU, disk and memory usage charts.

Right now sampling happens once a second, triggered by the heartbeat
event.That produces quite a bit of data for long builds, which will be
addressed in a separate commit by storing the data in a more compact
form.

(From OE-Core rev: 6f4e8180b5b4857eaf6caf410fd3a4a41ed85930)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:37:59 +00:00
Jair Gonzalez 5956492c20 parselogs: Whitelist GPT warnings as the device is fully functional
The warning occurs when the GPT image is not the same size than the
media into which it's being flashed, causing the backup GPT table
not being at the end of the disk. However, this is expected as the
image is created before having the information about the destination
media. The error is harmless, so it will be whitelisted.

Fixes [YOCTO 10481].

(From OE-Core rev: 5cc5cdc788308a79f8f0706e6d794c602ef427ed)

Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:37:59 +00:00
California Sullivan 23d2a7d156 parselogs.py: Whitelist iwlwifi firmware load error messages
The iwlwifi module of any given kernel has a minimum and maximum
supported firmware version. The kernel begins by attempting to load the
maximum version, and decrements until it is successful. The 4.8 kernel's
maximum supported firmware version is 24, but thus far only 22 has been
released, meaning we get errors for 24 and 23.

Filter out iwlwifi firmware load error messages, as they are not
necessarily indicative of real problems.

(From OE-Core rev: 7df570c2310efac8f9898da15deaac2b7df16655)

Signed-off-by: California Sullivan <california.l.sullivan@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:37:59 +00:00
Martin Vuille b7deb1f046 terminal.py: Pass string instead of bytes to ExecutionError to avoid exception
Based on run() in bitbake/lib/bb/process.py, ExecutionError() expects strings
not bytes. Passing bytes results in a "TypeError: Can't convert 'bytes' object
to str implicitly" exception.

Fixes Bug 10729

(From OE-Core rev: 063b63d4d324c23322ac1b6b7c7928e725d7b968)

Signed-off-by: Martin Vuille <jpmv27@yahoo.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:08 +00:00
Jose Perez Carranza b81477d329 runtime: Update test cases numbers for runtime tests
Update test case numbers on runtime tests to do match
with templates defined on Testopia for 2.3 release

(From OE-Core rev: d9df762b4c62b74f6d3a1521642ea86c26793a22)

Signed-off-by: Jose Perez Carranza <jose.perez.carranza@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:08 +00:00
Michael Wood cfdc5b853e oe-selftest: toaster Remove redundant Toaster test
This test has been ported to be run as part of Toaster's own tests.

(From OE-Core rev: f6366781a34dbdb8ec2d73f4fb36359bc15c4f42)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:06 +00:00
California Sullivan f3c9e66e8d parselogs: Whitelist NUC6 firmware load error message on genericx86-64
This was already whitelisted, but the 4.8 kernel changed the error
message, causing it to get caught by parselogs again.

Fixes [YOCTO #10494].

(From OE-Core rev: e1bad14231115f3b1a2bf844ef5b2022c648b55d)

Signed-off-by: California Sullivan <california.l.sullivan@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:14 +00:00
Joshua Lock 333d300fba lib/oe/lsb: better handle missing fields
Some rolling release distros, such as Arch Linux, don't include a
VERSION_ID field in their os-release file.

Change release_dict_osr() to better handle this optional field
being absent.

Further improve the resilience of the release_dict_*() methods by
always returning a dict and using dict.get() in distro_identifier()
to supply a default, empty string, value when then key is missing.

(From OE-Core rev: e36066dcc3b56cac1c695370ea178b566c0ebfd6)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:14 +00:00
Paul Eggleton 6e182aaa76 devtool: modify: support recipes with only local files as source
The hello-mod recipe is unusual in that it has only local files in
SRC_URI and builds these out of ${WORKDIR}. When you use devtool modify
on it, devtool puts all of those files in an "oe-local-files"
subdirectory of the source tree, which is not ${S} (or ${B}) any more
and thus building the recipe afterwards fails. It's a bit of a hack, but
symlink the files in oe-local-files into the source tree (and commit the
symlinks with an ignored commit so that the repo is clean) to work
around the problem. We only do this at time of extraction, so any files
added to or removed from oe-local-files after that won't be handled, but
I think there's a limit to how far we should go to support these kinds
of recipes - ultimately they are anomalies.

I initially tried a hacky workaround where I set effectively set B =
"${WORKDIR}" and that allowed it to build, but other things such as the
LIC_FILES_CHKSUM checks still broke because they expected to find files
in ${S}. Another hack where I set the sourcetree to point to the
oe-local-files subdirectory works for hello-mod but not for makedevs
since whilst that is similar, unlike hello-mod it does in fact have
files in the source tree (since it has a patch that adds COPYING) and
thus the same issue occurred.

Also tweak one of the tests that tries devtool modify / update-recipe on
the makedevs recipe to try building it since that would have caught this
issue.

Fixes [YOCTO #10616].

(From OE-Core rev: 857c06d6a1d161bf5a01311d07758bd4241929a3)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:14 +00:00
Stephano Cetola 93e8db2224 devshell: list commands when throwing NoSupportedTerminals
When attempting to run devshell, if no terminal is available, the
error being thrown was not very specific. This adds a list of
commands that failed, informing the user of what they can install to
fix the error.

[ YOCTO #10472]

(From OE-Core rev: c077f4aab2fc956408d4ad45c4e2e2ea6e480624)

Signed-off-by: Stephano Cetola <stephano.cetola@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:13 +00:00
Paul Eggleton 39fba4d70f oe-selftest: devtool: test that updating a file with subdir= works
If you have a file:// entry in SRC_URI with a subdir= parameter that
makes it extract into the source tree, then when you update that file in
oe-local-files and run devtool update-recipe then you want the original
file to be updated. This was made to work by OE-Core commit
9069fef5dad5a873c8a8f720f7bcbc7625556309 together with
31f1bbad248c36a8c86dde4ff57ce42efc664082, however until now there was no
oe-selftest test to verify it.

Note that in order to succeed this test also requires the fix
"lib/oe/recipeutils: ignore archives by default in
get_recipe_local_files()" since the test recipe uses a local tarball.

(From OE-Core rev: 936eba3e1059d1dcd5e58c1ce76870fff7b11b3c)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:13 +00:00
Paul Eggleton e3193b76e0 lib/oe/recipeutils: ignore archives by default in get_recipe_local_files()
By default, have get_recipe_local_files() not return any archive
files. This prevents a local tarball from being erroneously removed
from SRC_URI if you run "devtool modify" on a recipe followed by
"devtool update-recipe". It doesn't actually help you to directly
update the contents of such tarballs, but at least now it won't break
the recipe.

(From OE-Core rev: e9c418d4704c1bed4c5880e176e5288485f4f5a6)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:13 +00:00
Paul Eggleton 55a157f4e6 devtool: update-recipe: fix handling of compressed local patches
It is possible to use gzip or bzip2 to compress patches and still refer
to them in compressed form in the SRC_URI value within a recipe. If you
run "devtool modify" on such a recipe, make changes to the commit for
the patch and then run devtool update-recipe, we need to correctly
associate the commit back to the compressed patch file and re-compress
the patch, neither of which we were doing previously.

Additionally, add an oe-selftest test to ensure this doesn't regress in
future.

Fixes [YOCTO #8278].

(From OE-Core rev: e47d21624dfec6f71742b837e91da553f18a28c5)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:13 +00:00
Paul Eggleton d316363b7b lib/oe/patch: fix handling of patches with no header
If a patch applied by a recipe has no header and we turn the recipe's
source into a git tree (when PATCHTOOL = "git" or when using devtool
extract / modify / upgrade), the commit message ends up consisting only
of the original filename marker ("%% original patch: filename.patch").
When we come to do turn the commits back into a set of patches in
extractPatches(), this first line ends up in the "Subject: " part of
the file, but we were ignoring it because the line didn't start with the
marker text. The end result was we weren't able to get the original
patch name. Strip off any "Subject [PATCH x/y]" part before looking for
the marker text to fix.

This caused "devtool modify openssl" followed by "devtool update-recipe
openssl" (without any changes in-between) to remove version-script.patch
because that patch has no header and we weren't able to determine the
original filename.

(From OE-Core rev: d9971f5dc8eb7de551fd6f5e058fd24770ef5d78)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:12 +00:00
Paul Eggleton e5a391795a oe-selftest: devtool: test update-recipe with only local files
Add a test to ensure devtool update-recipe works properly on recipes
that contain only local files (since the other tests we have didn't test
that).

Relates to [YOCTO #10563].

(From OE-Core rev: bdc844b3f0c3fbddcd3523095899a5bd29797704)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:12 +00:00
Linus Wallgren b91e47fccb lib/oe/package_manager: .deb pre/postinst args
The debian policy manual and MaintainerScripts wiki page states that the
postinst script is supposed to be called with the `configure` argument
at first install, likewise the preinst script is supposed to be called
with the `install` argument on first install.

https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html
https://wiki.debian.org/MaintainerScripts

(From OE-Core rev: 3d9c3aae54589794ce3484fa1b21d1af2bd32661)

Signed-off-by: Linus Wallgren <linus.wallgren@scypho.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:11 +00:00
Joshua Lock 81afedc905 lib/oe/lsb: attempt to ensure consistent distro id regardless of source
The LSB Distributor ID and os-release NAME differ for most of the
distributions tested by the Yocto Project (CentOS, Debian, Fedora,
openSUSE and Ubuntu) however for all but openSUSE the os-release ID
matches the LSB Distributor ID when both are lowered before
comparison.

Therefore, in order to improve the consistency of identification of
a distribution, switch to using the os-release ID and converting
the ID value to lowercase.

Table showing comparison of LSB Distributor ID to os-release fields NAME
and ID for current Yocto Project supported host distributions:

Distribution | Version | Distributor ID   | NAME             | ID       |
-------------------------------------------------------------------------
CentOS       | 7       | CentOS           | CentOS Linux     | centos   |
Debian       | 8       | Debian           | Debian GNU/Linux | debian   |
Fedora       | 23      | Fedora           | Fedora           | fedora   |
Fedora       | 24      | Fedora           | Fedora           | fedora   |
openSUSE     | 13.2    | openSUSE project | openSUSE         | opensuse |
openSUSE     | 42.1    | SUSE LINUX       | openSUSE Leap    | opensuse |
Ubuntu       | 14.04   | Ubuntu           | Ubuntu           | ubuntu   |
Ubuntu       | 16.04   | Ubuntu           | Ubuntu           | ubuntu   |

[YOCTO #10591]

(From OE-Core rev: 8689e5618d45c2119134ea64754430c06a93ea09)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-15 15:19:54 +00:00
Joshua Lock 42441ea481 lib/oe/lsb: prefer /etc/os-release for distribution data
os-release(5) is an increasingly standard source of operating system
identification and more likely to be present on modern OS deployments, i.e.
many container variants of common distros include os-release and not the
lsb_release tool.

Therefore we should favour parsing /etc/os-release in distro_identifier(),
try lsb_release when that fails and finally fall back on various distro
specific sources of OS identification.

(From OE-Core rev: fc4eddecddec68d03a985086fa32db40ad0c7bfc)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-15 15:19:54 +00:00
Joshua Lock 545f5f96d9 lib/oe/lsb: make the release dict keys consistent regardless of source
Rather than have the distro_identifier method look for different keys in
the dict depending on the source ensure that each function for retrieving
release data uses the same key names in the returned dict.

(From OE-Core rev: 2ddd6ddaf0c5ba14ae83347eba877ac9ef179c76)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-15 15:19:54 +00:00
Ross Burton 025f917cec lib/oe/qa: handle binaries with segments outside the first 4kb
The ELF parser was assuming that the segment tables are in the first 4kb of the
binary.  Whilst this generally appears to be the case, there have been instances
where the segment table is elsewhere (offset 2MB, in this sample I have).  Solve
this problem by mmap()ing the file instead.

Also clean up the code a little whilst chasing the problem.

(From OE-Core rev: a66660aa5bb709547ce0b65a4563e4217c3c3d9f)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-15 15:19:53 +00:00
Joshua Lock 5193dfdb1f lib/oe/path: remove duplicate import
There's no need to import glob inside copyhardlinktree() as it's
already imported for the entire path module.

(From OE-Core rev: 42dc4695da136a15bebb7525b1da5c2722b10a28)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-15 15:19:53 +00:00
Ross Burton 4e6c5d8769 distro_check: MeeGo is long dead, compare against Clear Linux instead
Instead of checking against a file that represents a distribution that hasn't
existed for years, fetch package names for Clear Linux instead.

[ YOCTO #10601 ]

(From OE-Core rev: 006c4db0974c42ff0f6950dd24e61c008f801679)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-15 15:19:53 +00:00
Ross Burton b654f5175d Revert "oeqa/selftest/kernel.py: Add new file destined for kernel related tests"
breaking on selftest

NOTE: recipe core-image-minimal-1.0-r0: task do_rootfs: Started
ERROR: core-image-minimal-1.0-r0 do_rootfs: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:license_create_manifest(d)
     0003:
File: '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/classes/license.bbclass', lineno: 48, function: license_create_manifest
     0044:    pkg_dic = {}
     0045:    for pkg in sorted(image_list_installed_packages(d)):
     0046:        pkg_info = os.path.join(d.getVar('PKGDATA_DIR', True),
     0047:                                'runtime-reverse', pkg)
 *** 0048:        pkg_name = os.path.basename(os.readlink(pkg_info))
     0049:
     0050:        pkg_dic[pkg_name] = oe.packagedata.read_pkgdatafile(pkg_info)
     0051:        if not "LICENSE" in pkg_dic[pkg_name].keys():
     0052:            pkg_lic_name = "LICENSE_" + pkg_name
Exception: FileNotFoundError: [Errno 2] No such file or directory: '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/sysroots/qemux86-64/pkgdata/runtime-reverse/kernel-4.8.3-yocto-standard'

This reverts commit c3d2df883a.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-15 15:19:52 +00:00
Paul Eggleton 43e652f3d1 devtool: add "rename" subcommand
When you run devtool add on a source tree we attempt to figure out the
correct name and version for the recipe. However, despite our best
efforts, sometimes the name and/or version we come up with isn't
correct, and the only way to remedy that up until now was to reset the
recipe, delete the source tree and start again, specifying the name this
time. To avoid this slightly painful procedure, add a "rename"
subcommand that lets you rename the recipe and/or change the version.

(From OE-Core rev: 9303d8055c45a0f6af295d70a6f6a8b9d8d8a7c9)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-07 11:04:22 +00:00
Paul Eggleton 7aebaa4204 oe-selftest: devtool: fix error message in _test_recipe_contents()
If a variable is being set in the recipe when we've explicitly passed
None as the value to _test_recipe_contents() indicating that it
shouldn't be set at all, then we should be printing out the variable
name in the assertion message but it seems like I forgot to do a
substitution. Also include the value for informational purposes.

(From OE-Core rev: 0dafcb158003fb13f82c266f607d9967fca321db)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-07 11:04:22 +00:00
Paul Eggleton e0b62c38ac recipetool: create: separate LICENSE items with & by default
recipetool sets the LICENSE value based on licenses detected from the
source tree. If there are multiple licenses then they were being
separated by spaces, but this isn't actually legal formatting and if
you're using "devtool add" you get a warning printed when devtool
parses the recipe internally.

Earlier I had made a conscious decision to do it this way since it's up
to the user to figure out whether the multiple licenses should all apply
(in which case they'd be separated with &) or if there is a choice of
license (in which case | is the correct separator). However, I've come
to the conclusion that we can just default to & and then the ugly
warning goes away, and it's the safest alternative of the two (and most
likely to be correct, since it's more common to have a codebase which is
made up of code with different licenses, i.e. all of them apply to the
combined work).

I've tweaked the comment that we add to the recipe to explicitly state
that we've used & and that the user needs to change that if that's not
accurate.

Fixes [YOCTO #10413].

(From OE-Core rev: ecac6aee8cf3313350b58c21012bcd67cfb915e4)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-07 11:04:22 +00:00
Ross Burton 1655d55764 distro_check: partial rewrite to make it work again
This library suffered as part of the Python 2 to Python 3 migration and stopped
working entirely.

Fix all the migration problems such as files being treated as strings but opened
in binary mode, insufficient use of with on files, and so on.

Rewrite large amounts to be Pythonic instead of C-in-Python.

Update OpenSuse and Fedora URLs.

Fedora now splits the archive alphabetically so handle that.

[ YOCTO #10562 ]

(From OE-Core rev: 58de12eaaac9c60bb8fc84a3a965ef86d2a39ae0)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-06 23:35:36 +00:00
Paul Eggleton 385f108377 lib/oe/recipeutils: print just filename in bbappend_recipe() if in temp dir
If you use devtool update-recipe with the --append option, and a "local"
(in oe-local-files) has been modified we copy it into the specified
destination layer. With the way the devtool update-recipe code works now
the source is always a temp directory, and printing paths from within
that is just confusing, so if the path starts with the temp directory
then just print the file name alone.

(From OE-Core rev: 61475f0267d40c618ebf36023d0b6414a25975cb)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-06 23:35:36 +00:00
Samuli Piippo b18901d9bc package_manager.py: correctly remove all dependent packages
Do not use --force-depends when trying to remove all dependent packages,
as it removes only the selected package and not the dependent packages.

(From OE-Core rev: a82e8725902086dab785a0b14305927dae1e4e8d)

Signed-off-by: Samuli Piippo <samuli.piippo@qt.io>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-06 23:35:33 +00:00
Robert Yang d31d1ad4e5 oe/copy_buildsystem.py: dereference symlink
When there is a relative symlink in the layer, for example:
symA -> ../out/of/layer/file

symA will be invalid fater copied, it would be invalid from build time
if it points to a relative path, and would be invalid after extracted
the sdk if it points to a absolute py. Dereference symlink when copy
will fix the problem.

Use tar rather than shutil.copytree() to copy is because:
1) shutil.copytree(symlinks=Fasle) has bugs when dereference symlinks:
   https://bugs.python.org/issue21697
   And Ubunutu 1404 doesn't upgrade python3 to fix the problem.

2) shutil.copytree(symlinks=False) raises errors when there is a invalid
   symlink, and tar just prints a warning, tar is preferred here since
   the real world is unpredicatable

3) tar is faster than shutil.copytree() as said by oe.path.copytree()

So use tar to copy.

(From OE-Core rev: f4d70bb0882eec4fb46cd942f2796fad57c72982)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-06 23:35:32 +00:00
Costin Constantin c3d2df883a oeqa/selftest/kernel.py: Add new file destined for kernel related tests
[YP#7202]:  Test for linux-dummy
The new kernel.py file is intended for kernel related test cases.
The test for linux-dummy will ensure it is in good shape and can
be used as a kernel replacement at build time. To do this, the
test will first clean sstate for linux-dummy target, ensuring no
file is present in the stamps directory. After, core-image-minimal
is built, ensuring linux-dummy can be used as a kernel substitute.

(From OE-Core rev: 98c6ebf1e05158c689e01b785d32757847cdb10c)

Signed-off-by: Costin Constantin <costin.c.constantin@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-01 10:05:46 +00:00
Davis, Michael e66f5eabed terminal.py: Add compatiblity for konsole 16.08.1
Konsole has dropped support for the nofork flag.  It has been replaced with the seperate flag.

(From OE-Core rev: f0b193b63d4c468c3aa58e15ef5a991e04b9b9a2)

Signed-off-by: Michael Davis <michael.davis@essvote.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-28 16:15:18 +01:00
Joshua Lock 751cb2c894 Remove RM_OLD_IMAGE, it's no longer useful
Since the move to put image deployment under sstate control in
d54339d4b1a7e884de636f6325ca60409ebd95ff old images are automatically
removed before a new image is deployed (the default behaviour of the
sstate logic).

RM_OLD_IMAGE is therefore no longer required to provide this
behaviour, remove the variable and its users.

(From OE-Core rev: 93631befe8b962bf99524746b49f4ebca336175c)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-15 10:01:42 +01:00
Ross Burton 84198174bf lib/oe/qa: add ELF machine to string function
Add a function (and test suite) to turn the ELF machine field (e_machine) into a
string, so we can tell the user "x86-64" instead of 0x3E.

(From OE-Core rev: 72336003741fb16a7ecdd6b753eae56310413ff7)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-11 22:19:21 +01:00
Richard Purdie 78c01995e3 oeqa/sshcontrol: Handle interrupted system call error
Deal with an interrupted system call gracefully:

|   File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-systemd/build/meta/lib/oeqa/utils/sshcontrol.py", line 55, in _run
|     if select.select([self.process.stdout], [], [], 5)[0] != []:
| InterruptedError: [Errno 4] Interrupted system call

(From OE-Core rev: 556125e4004fb7ac5169b59f51dc151f18c1806a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-07 16:43:58 +01:00
Ed Bartosh 09a9de45e1 wic: selftest: add test for sdimage-bootpart
Test creation of sdimage-bootpart image

(From OE-Core rev: 8b18568591f66aa9770798bc61123898de92b8a9)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-07 16:43:58 +01:00
Ed Bartosh bed4a13440 wic: selftest: add test for systemd-bootdisk
Test creation of systemd-bootdisk image.

(From OE-Core rev: b8eef88536b944ea35664b9a6a9496fb0a3f7988)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-07 16:43:58 +01:00
Peter Kjellerstedt d067b0e1e4 package_manager.py: Allow multiple regexps in PACKAGE_EXCLUDE_COMPLEMENTARY
The PACKAGE_EXCLUDE_COMPLEMENTARY variable can currently only contain
one regular expression. This makes it hard to add to it from different
configuration files and recipes.

Allowing it to contain multiple, whitespace separated regular
expressions should be backwards compatible as it is assumed that
whitespace is not used in package names and thus is not used in any
existing instances of the variable.

After this change, the following three examples should be equivalent:

  PACKAGE_EXCLUDE_COMPLEMENTARY = "foo|bar"

  PACKAGE_EXCLUDE_COMPLEMENTARY = "foo bar"

  PACKAGE_EXCLUDE_COMPLEMENTARY = "foo"
  PACKAGE_EXCLUDE_COMPLEMENTARY += "bar"

(From OE-Core rev: a5f7e98a94e96d40b1276c85249619aa8d7be847)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-06 07:51:00 +01:00
Peter Kjellerstedt 2334201960 package_manager.py: Allow a leading - in PACKAGE_EXCLUDE_COMPLEMENTARY
This allows a regular expression specified in
PACKAGE_EXCLUDE_COMPLEMENTARY to have a leading dash. Without this,
the dash was treated by oe-pkgdata-util as the beginning of a command
line argument. E.g., if PACKAGE_EXCLUDE_COMPLEMENTARY = "-foo$", it
resulted in an error like:

  ERROR: <imagename>-1.0-r0 do_populate_sdk: Could not compute
  complementary packages list. Command '<topdir>/scripts/oe-pkgdata-util -p
  <builddir>/tmp/sysroots/<machine>/pkgdata glob
  <workdir>/installed_pkgs.txt *-dev *-dbg -x -foo$' returned 2:
  ERROR: argument -x/--exclude: expected one argument
  usage: oe-pkgdata-util glob [-h] [-x EXCLUDE] pkglistfile glob [glob ...]

(From OE-Core rev: ac4ca41d3a27356d46c0c39053e74d3519b24c44)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-06 07:51:00 +01:00
Markus Lehtonen b16001f3e2 oeqa.buildperf: measure apparent size instead of real disk usage
This change aligns disk usage measurements of the eSDK test with the old
build-perf-test.sh script. And thus, also makes the results between the
old and the new script comparable.

(From OE-Core rev: dadb84936b3672dcf07e5ab8226158136762801f)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-06 07:51:00 +01:00
California Sullivan ed12917d59 parselogs.py: Add disabling eDP error to x86_common whitelist
The NUC6 firmware tells the kernel to try and initialize an embedded
DisplayPort it does not have, causing this warning. Its harmless, so
just whitelist it.

Fixes [YOCTO #9434].

(From OE-Core rev: 4c3fb7f63aad4a5d1b9720c76091cd0646859c2a)

Signed-off-by: California Sullivan <california.l.sullivan@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-05 10:10:11 +01:00
Benjamin Esquivel d89e16d5cc oeqa/selftest: Update test after fetcher error changes
The following poky commit:

4359ef08 base.bbclass: Use bb.fatal() instead of raising FuncFailed

changed the way the fetcher error is reported.

Previous reporting:
...Function failed: Fetcher failure for URL:...

New reporting:
...Fetcher failure for URL:...

Updating how the check is done fixes the test error and accurately
confirms the tested scenario for test_invalid_recipe_src_uri.

[YOCTO #10370]

(From OE-Core rev: 197da17dc97cef87375ae9190c6d1495e1c615b9)

Signed-off-by: Benjamin Esquivel <benjamin.esquivel@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-05 10:10:10 +01:00
Richard Purdie 2d7df9be8d oeqa/sstatetests: Add test for multilib allarch checksums
Switching between multilib configurations should not change allarch recipe
or nativesdk checksums. Add a new sstate test for this based on the standard
allarch test.

(From OE-Core rev: 660543601171f88c75fb4e90f34dac86037f3f23)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-01 21:45:57 +01:00
Francisco Pedraza 7ae326c76b oeqa/utils: Add StreamHandler to logger
StreamHandler was added due missing log information on the console in
oe-selftest with Qemu Runner

(From OE-Core rev: a4e2df151af781edbcb6b0e17b51b5ed226bf77f)

Signed-off-by: Francisco Pedraza <francisco.j.pedraza.gonzalez@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-01 21:45:57 +01:00
Jianxun Zhang e0e8cd039c parselogs: Update uvesafb errors in qemu whitelist
This change only whitelists the timeout message in infinite
wait case in uvesafb driver.

With the latest timeout patch introducing infinite wait in
uvesafb driver, we whitelist the timeout message since it works
as a warning for issues related to timeout not to be fixed in
build servers.

We remove other errors for bug-discovering purposes in some cases
where these lines are still worthy to be caught (not whitelisted):

The removed errors show up again in the infinite wait case. It
indicates a different root cause or the timeout patch doesn't work
correctly.

Timeout happens when developers explicitly set a non-negative timeout
of a limited period to wait for task completion in uvesafb driver.

Timeout or/and errors occur when kernel doesn't have the latest
timeout patch in driver.

Note: The latest timeout patch is tracked by:
a2966330bc

[YOCTO #8245]

(From OE-Core rev: 2e15b478343c6703c37b9a45e61c9de200d98027)

Signed-off-by: Jianxun Zhang <jianxun.zhang@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-30 16:51:15 +01:00
Richard Purdie 12905346a0 oeqa/sstatetests: Ensure we cover deb packaging backend for sstate test
Currently we weren't testing the deb backaned for sstate correctness
and there was a bug that had crept in. Ensure we cover all package
backends with the test regardless of what the distro/conf sets.

(From OE-Core rev: cdaafc3729700778d95afc2413553d7b41c1317b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-28 10:16:04 +01:00
Jussi Kukkonen 46c5e7adb8 lib/oeqa: Regenerate galculator configure
galculator configure seems to be so old it does not
recognise --with-libtool-sysroot: regenerate configure.

Fixes [YOCTO #10191].

(From OE-Core rev: 3e838773462e77cb2e3ba9a69534260f89ca4904)

Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-28 10:16:04 +01:00
California Sullivan bb744b82c7 parselogs.py: Add amd_nb error to x86_common whitelist
This has always silently failed on hardware without AMD Northbridge,
and a recent kernel patch made it not silent. It would be ideal to only
whitelist the error for genericx86 MACHINEs and disable the CONFIG
option that enables it in intel-* MACHINEs, but in order to disable
this configuration option we would have to enable EXPERT and
DEBUG_KERNEL, which we don't want. Instead just whitelist it on all
x86 MACHINEs.

Fixes [YOCTO #10261].

(From OE-Core rev: 9c432dae1045a087f8eb2de7c9bd3a9cbd46c459)

Signed-off-by: California Sullivan <california.l.sullivan@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-28 10:16:03 +01:00
Paul Eggleton 7caf628e83 lib/oe/patch: improve accuracy of patch header extraction
When PATCHTOOL = "git", if we need to manually apply a patch and then
commit it (i.e. when git am doesn't work) we try to extract the author /
date / shortlog from the patch header. Make the following improvements
to that extraction process:

* If there's no explicit Subject: but the first line is followed by a
  blank line, isn't an Upstream-Status: or Index: marker and isn't too
  long, then assume it's good enough to be the shortlog. This avoids
  having too many patches with "Upgrade to version x.y" as the shortlog
  (since that is often when patches get added).
* Add --follow to the command we use to find the commit that added the
  patch, so we mostly get the commit that added the patch rather than
  getting stuck on upgrade commits that last moved/renamed the patch
* Populate the date from the commit that added the patch if we were able
  to get the author but not the date from the patch (otherwise you get
  today's date which is less useful).

(From OE-Core rev: 896cfb10ec166a677cbb3b4f8643719cabeb7663)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-24 07:30:09 +01:00
Paul Eggleton f6928407ed lib/oe/patch: exclude "From <hash>" from commit message when PATCHTOOL is "git"
If you leave "From <hash>" lines in the commit message it can actually
break git rebase because it tries to interpret the line in the context
of the current repository, and if the hash is invalid then a rebase
will blow up with:

  fatal: git cat-file: could not get object info

or in newer git versions:

  error: unable to find <hash>
  fatal: git cat-file <hash>: bad file

(I hit this when I tried to do a devtool upgrade on openssl to 1.0.2i
the first time I did "git rebase --skip")

(From OE-Core rev: 19a6b18ac23cb2d7bb89203f774b2bee7f0cb03c)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-24 07:30:09 +01:00
Jose Lamego c902e7b937 oeqa/selftest/base: backup and restore local configuration files
Selftests' cleanup method during test setup is not capable of
restoring local configuration files that remain modified after
aborting a test through a keyboard interruption.
This change creates backups for local.conf and bblayers.conf at
test setup, restore them when found, and deletes them at cleanup.

[YOCTO #9390]

(From OE-Core rev: 0877278e07e4c2494c4c23199490dc47a5cee69d)

Signed-off-by: Jose Lamego <jose.a.lamego@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-23 14:56:39 +01:00
Ross Burton a767c78abe oeqa/selftest/lic-checksum: don't report the expected failure to errors.yp
This test has a bitbake invocation that is expected to fail, so inhibit
report-error running.

(From OE-Core rev: b2771e17a5f301423f65be9f93c9c1b1e7f8ab93)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-20 15:11:09 +01:00
Aníbal Limón a8cf594bce oeqa/runtime/parselogs.py: Add ignore of tsc calibration fail in x86
We are experimenting failures to calibrate CPU's using TSC in x86
VM's due to usage of nested KVM [1], this is a known issue [2][3]
in virtualization environments, for detail explnation see [4].

Also we already have an ignore for 'TSC Fast calibration fail'.

[1] http://errors.yoctoproject.org/Errors/Details/83684/
[2] https://bugzilla.redhat.com/show_bug.cgi?id=814231
[3] https://lists.nongnu.org/archive/html/qemu-devel/2010-09/msg01547.html
[4] https://www.kernel.org/doc/Documentation/virtual/kvm/timekeeping.txt

(From OE-Core rev: 2271f59a0f506f89f9fea6777701c4b40790ddd9)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-20 15:11:09 +01:00
Mark Hatle 0a04692279 package_manager.py: Change diagnostic messages per IRC
Based on a discussion with IRC user: Ulfalizer

It was suggested that removing the diagnostic list, and replacing it with a
simple hint to what might be causing the problem was a better solution.

(From OE-Core rev: ca78313665b23bd7fee85f034acfe1eb1009bd65)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-20 15:11:08 +01:00
Mark Hatle f1e3dc029f package_manager.py: Adjust error message order
Move the debug before the error (as it can take many pages.)  This makes it
much easier for the user to see the actual error message as it is still on
the screen.

(From OE-Core rev: d643fb2a9cb5bd0d8b0105e9d44b989a49ffa963)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-20 15:11:08 +01:00
Paul Eggleton ce1904a160 lib/oe/recipeutils: fix invalid character detection in validate_pn()
* validate_pn() is supposed to protect against invalid characters, fix
  the function so that it actually does (unanchored regex strikes
  again...)
* However, now that the function is enforcing the restrictions, we do
  still want to allow + in recipe names (e.g. "gtk+")

(From OE-Core rev: c5d5a1baf98a11676537fb5e9f8ec4409e30c1fd)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-20 15:11:07 +01:00
Hongxu Jia 06e56abbd2 package_manager.py: fix bitbake package-index failed
Previously the following commit in oe-core move RPM metadata
from DEPLOY_DIR to WORKDIR.
-----------
commit a92c196449c516fe51786d429078bbb1213bb029
Author: Stephano Cetola <stephano.cetola@linux.intel.com>
Date:   Wed Aug 10 13:03:16 2016 -0700

    Allow for simultaneous do_rootfs tasks with rpm

    Give each rootfs its own RPM channel to use.  This puts the RPM metadata
    in a private subdirectory of $WORKDIR, rather than living in DEPLOY_DIR
    where other tasks may race with it.
-----------

In the modification of 'class RpmIndexer, it should not
directly set arch_dir with WORKDIR. It caused 'bitbake
package-index' could not work correctly.

Assign WORKDIR as input parameter at RpmIndexer initial time
could fix the issue.

(From OE-Core rev: 3c8c8501d0a19b566a94a9e06afe40642b444958)

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-16 15:24:02 +01:00
Paul Eggleton 7ba62d65d2 oeqa/sdkext/devtool: use a smaller module to test node.js functionality
The "forever" package, despite its innocent description, actually drags
in a surprising number of dependencies and as a result the nodejs test
takes up to 10 minutes as a result. Pick a different example with a much
more reasonable set of dependencies.

Addresses part of [YOCTO #10254].

(From OE-Core rev: 638ee71da967f093071ba25e0ea5c467dab65339)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-16 15:24:02 +01:00
bavery 92862c678f qemurunner: print out the runqemu command
This logs the launch command that was used for runqemu while running -c
testimage.  This way, if I'd like to easily launch qemu manually in
order to debug a failed test, I know what commmand was run to create
the qemu instance.

(From OE-Core rev: 34aa20c6f323bbf7ad53beb643126e4e03634708)

Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-16 15:24:02 +01:00
Ross Burton a7a6d05579 oeqa/oetest: show stderr when running commands
To help debug failures, redirect stderr to stdout in oeSDKTest.run() and
oeSDKExtTest.run().

(From OE-Core rev: 4cd143e0de7f0082f60f273f442f6255f28ec3e7)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-16 15:24:02 +01:00
Aníbal Limón 62c45ffbc7 oeqa/utils/decorators: LogResults fix race condition in linkfile
In order to avoid race condition when test if exists the linkfile
use bb.utils.lock, the best solution is to create a unique name
for the link file.

There is no way to create a unique linkfile name at this decorator
because is needed the machine and image variables, those variables
can't be passed easily in this code.

To avoid broke test export functionality use a try/except because bb
isn't available when use test export

[YOCTO #10225]

(From OE-Core rev: 059d475b6bce1e5414170a4fe2e7989f6b0eacd6)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-15 12:15:07 +01:00
Richard Purdie 56a4900273 Revert "oeqa.runtime.smart: work around smart race issues"
We have a proper fix now so we can remove the workaround.

This reverts commit 4d268abc2fc892c5d34449f78c8e9f2b1a9d6bac.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-14 22:22:13 +01:00
Paul Eggleton 3f95a21c71 lib/oe/copy_buildsystem: fix building eSDK with indirect paths in BBLAYERS
Indirect paths (e.g. ${TOPDIR}/../meta-something) do generally work if
used in BBLAYERS in bblayers.conf. However, if you built an extensible
SDK with this configuration then the creation of the workspace within
the SDK using devtool in do_populate_sdk_ext failed. This is because
the copy_buildsystem code was no longer correctly recognising that the
core layer ("meta") was part of a repository (e.g. openembedded-core /
poky) that should be shipped together - because of the indirection - and
thus it was splitting out the meta directory, and a number of places in
the code assume that the meta directory is next to the scripts
directory. Use os.path.abspath() to flatten out any indirections.

(From OE-Core rev: 7c0788cd2390fd0e1ec84bc9dbebcb67daee429f)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-14 22:22:13 +01:00
Robert Yang b8e2bab119 meta: cleanup d.getVar(var, 1)
(From OE-Core rev: 79fe476be233015c1c90e9c3fb4572267b5551d1)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-14 22:22:07 +01:00
Edwin Plauchu 2c7849fdcb oeqa: Remove linux user utilized for rpm test.
When trying to re-test smart rpm tests. A fail arises
due to a linux user previously created upon the image.
We've added a few lines to delete such user and his home dir
when finishing test.

[YOCTO #9204]

(From OE-Core rev: d1a80ac434bb798634bbb792ff7df59148ec26be)

Signed-off-by: Edwin Plauchu <edwin.plauchu.camacho@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-14 22:22:07 +01:00
Markus Lehtonen 0736079dc6 oeqa.buildperf: another fix for splitting 'nevr' string
When processing buildstats we determine recipe name, epoch, version and
revision from the per-recipe buildstat directory name.  One previous
patch made an assumption that package version starts with a number.
That might not be true because of a packaging mistake or whatever
reason. Thus, if a version starting with a number is not found, fall
back to the "old" method of just taking the second-last dash-delimited
part (the one before revision).

(From OE-Core rev: 936eb9aac055f1d4f41bb351477a0f5bebf76a0e)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-14 22:22:07 +01:00
Richard Purdie 996aaf93ff oeqa/parselogs: Don't use cwd for file transfers
If you run:

MACHINE=A bitbake <image> -c testimage
MACHINE=B bitbake <image> -c testimage

and A has errors in parselogs, machine B can pick these up and cause
immense confusion. This is because the test transfers the log files
to cwd which is usually TOPDIR. This is clearly bad and this patch
uses a subdir of WORKDIR to ensure machines don't contaminate each
other.

Also ensure any previous logs are cleaned up from any existing
transfer directory.

(From OE-Core rev: ac8f1e58ca3a0945795087cad9443be3e3e6ead8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-14 22:22:06 +01:00
Markus Lehtonen 68e35c8e37 oeqa.runtime.smart: work around smart race issues
Yucku hack around test failures which ultimately are caused by a race in
smartpm itself. Issuing smartpm commands in quick succession causes
races in package cache of smartpm on some systems. This patch mitigates
the problem by sleeping for 1 second after each smartpm command that
modifies the system.

[YOCTO #10244]

(From OE-Core rev: 4d268abc2fc892c5d34449f78c8e9f2b1a9d6bac)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-09 12:12:23 +01:00
Richard Purdie d3991342ed oeqa/runtime/smart: Prune feeds to save memory
Full package feed indexes overload a 256MB image so reduce the number of rpms
the feed. Filter to p* since we use the psplash packages and this leaves some
allarch and machine arch packages too.

[YOCTO #8771]

(From OE-Core rev: f352c3b71cbf50846c7de31046202296b38713cc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-09 12:12:23 +01:00
Richard Purdie 8c46605d16 oeqa: Use snapshot instead of copying the rootfs image
Rather than copying images, use the snapshot option to qemu. This fixes a regression
caused by the recent runqemu changes where the wrong images were being testes since
the image is copied without the qemuboot.conf file. This means the latest image is
found by runqemu rather than the specified one, leading to various confused testing
results.

It could be fixed by copying more files but use snapshot mode instead.

(From OE-Core rev: eab91997d415b0e690b3482749a32087e6a8b00a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-09 12:07:32 +01:00
Robert Yang 95331c6d4e qemurunner.py/qemutinyrunner.py: remove runqemu-internal
There is no runqemu-internal any more.

(From OE-Core rev: 14bacf7203ab7a638b67eb143225d8c75bbb703d)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-09 12:07:32 +01:00
Robert Yang af7ec7afc0 parselogs: Whitelist qemux86 error message with qemu 2.7.0
qemu 2.7.0 introduces kernel errors:

[    2.310768] pci 0000:00:00.0: [11ab:4620] type 00 class 0x060000
[    2.311338] pci 0000:00:00.0: [Firmware Bug]: reg 0x14: invalid BAR (can't size)
[    2.311604] pci 0000:00:00.0: [Firmware Bug]: reg 0x18: invalid BAR (can't size)
[    2.311835] pci 0000:00:00.0: [Firmware Bug]: reg 0x1c: invalid BAR (can't size)
[    2.312063] pci 0000:00:00.0: [Firmware Bug]: reg 0x20: invalid BAR (can't size)
[    2.312323] pci 0000:00:00.0: [Firmware Bug]: reg 0x24: invalid BAR (can't size)
[    2.314320] pci 0000:00:0a.0: [8086:7110] type 00 class 0x060100
[    2.315363] pci 0000:00:0a.1: [8086:7111] type 00 class 0x010180

Whitelist this for now since this is preferable to the random failures
we're seeing from qemuppc with 2.6.0.

(From OE-Core rev: 4d542cdc86c34f0f4a3dde8b0aab059bca76a9fb)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-09 11:53:37 +01:00
Richard Purdie e399ca1dd8 parselogs: Whitelist qemuppc error message with qemu 2.7.0
qemu 2.7.0 introduces kernel errors:

[0.474981] pci 0000:00:0f.0: reg 0x10: [io  0x0400-0x041f]
[0.483796] pci 0000:00:0f.0: reg 0x20: can't handle BAR above 4GB (bus address 0xffffffff82800000)
[0.484204] pci 0000:00:0f.0: reg 0x20: [mem size 0x00800000 64bit pref]
[0.488077] pci 0000:00:0f.0: reg 0x30: [mem 0x83000000-0x8303ffff pref]
[0.488903] pci 0000:00:10.0: [1af4:1005] type 00 class 0x00ff00
[0.490485] pci 0000:00:10.0: reg 0x10: [io  0x0480-0x049f]
[0.496512] pci 0000:00:10.0: reg 0x20: can't handle BAR above 4GB (bus address 0xffffffff83800000)
[0.496783] pci 0000:00:10.0: reg 0x20: [mem size 0x00800000 64bit pref]
[0.500345] pci 0000:00:11.0: [1af4:1001] type 00 class 0x010000
[0.501790] pci 0000:00:11.0: reg 0x10: [io  0x0500-0x053f]
[0.507362] pci 0000:00:11.0: reg 0x20: can't handle BAR above 4GB (bus address 0xffffffff84000000)
[0.507677] pci 0000:00:11.0: reg 0x20: [mem size 0x00800000 64bit pref]
[0.513905] pci_bus 0000:00: busn_res: [bus 00-ff] end is updated to 00
[0.516493] PCI 0000:00 Cannot reserve Legacy IO [io  0x0000-0x0fff]
[0.517512] pci 0000:00:0f.0: BAR 4: assigned [mem 0x80800000-0x80ffffff 64bit pref]
[0.518877] pci 0000:00:10.0: BAR 4: assigned [mem 0x82800000-0x82ffffff 64bit pref]
[0.519890] pci 0000:00:11.0: BAR 4: assigned [mem 0x83800000-0x83ffffff 64bit pref]

Whitelist this for now since this is preferable to the random failures
we're seeing from qemuppc with 2.6.0.

(From OE-Core rev: 2472ed5393ab01ad79c011ea19c73224ed5125de)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-09 11:53:37 +01:00
Paul Eggleton 39d3aa2828 devtool: update-recipe: support files with subdir=
It's rare but there are recipes that have individual files (as opposed
to archives) in SRC_URI using subdir= to put them under the source tree,
the examples in OE-Core being bzip2 and openssl. This broke devtool
update-recipe (and devtool finish) because the file wasn't unpacked into
the oe-local-files directory and thus when it came time to update the
recipe, the file was assumed to have been deleted by the user and thus
the file was erroneously removed. Add logic to handle these properly so
that this doesn't happen.

(We still have another potential problem in that these files become part
of the initial commit from upstream, which could be confusing because
they didn't come from there - but that's a separate issue and not one
that is trivially solved.)

(From OE-Core rev: 9069fef5dad5a873c8a8f720f7bcbc7625556309)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-08 00:36:48 +01:00
Paul Eggleton 94aefd9a39 lib/oe/patch: handle non-UTF8 encoding when reading patches
When extracting patches from a git repository with PATCHTOOL = "git" we
cannot assume that all patches will be UTF-8 formatted, so as with other
places in this module, try latin-1 if utf-8 fails.

This fixes UnicodeDecodeError running devtool update-recipe or devtool
finish on the openssl recipe.

(From OE-Core rev: 579e4d54a212d04cfece2c9fc0635d7ac1644058)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-08 00:36:48 +01:00
Markus Lehtonen 98264a9191 oeqa.buildperf: be sure to use the latest buildstats
Be sure to take the latest buildstats if multiple buildstats are found.

(From OE-Core rev: bad495f0d0144728a0132c3d3c4d98c24ead4afd)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-08 00:32:43 +01:00
Markus Lehtonen d021889ba9 oeqa.buildperf: try harder when splitting 'nevr' string
Try to be more intelligent when splitting out recipe name, epoch,
version and revision from the buildstat directory name. Previous
assumption was that package versions never contain a dash but obviously
that is not necessarily true. The new assumption is that the package
version starts with a number.

(From OE-Core rev: 91d3fce1eb3e27d646afba8cf3c03ae560412d1d)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-08 00:32:43 +01:00
Markus Lehtonen f7ca989ddc oeqa.buildperf: correct globalres time format
Always use two digits for (integer part of) seconds, i.e. show '1:02.34'
instead of '1:2.34'.

(From OE-Core rev: 55bb6816aca39bfa25d4f7e2158a57a5f0ac1cca)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-06 10:24:04 +01:00
Joshua Lock e8e81789f9 selftest/liboe: add a test for copyhardlinktree()
Add a simple test to validate that the number of files in the
destination matches the number of files in the source after the
copyhardlinktree() has been performed.

(From OE-Core rev: ca5c718b309524e46818627f8b5c9260d009472d)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-06 10:24:04 +01:00
Joshua Lock 822c708e8f oe.path: fix copyhardlinktree()
The change to preserve extended attributes in copytree() and
copyhardlinktree() (e591d69103a40ec4f76d1132a6039d9cb1555103)
resulted in an incorrect cp invocation in copyhardlinktree() when
the source directory contained hidden files.

This was because the passed src was modified in place but some code
paths expected it to remain unmodified from the passed value.
Resolve the issue by constructing a new source string, rather than
modifying the passed in string.

(From OE-Core rev: 2b9fdd8448c2c29418d1c3fca9fe1789466f09b4)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-06 10:24:04 +01:00
Richard Purdie 42e2b97441 oeqa/parselogs: Add qemuarm64 warning from 4.8 kernel to whitelist
(From OE-Core rev: ae865fee26d2a32ae07236fc7aa1cf1b234a2156)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-05 11:56:01 +01:00
Bruce Ashfield d9c2c02e96 oeqa/parselogs/qemuarm: Whitelist amba and jitter for 4.8+ kernels
With the update to the 4.8 kernel the versatile platform (and hence
qemuarm) has switched to a device tree boot.

We are using an ummodified mainline kernel versatilepb device tree,
which includes definitions of multiple amba devices. These devices
are not present in the qemu system emulation, hence throw warnings
during boot.

These warnings are not unique to oe-core, and rather than carry kernel
patches to the device tree (for now), we whitelist the known warnings
so qa testing will pass. We also can't turn amba off completely, since
it is providing valid devices (like the serial port) and AMBA is
force selected by other kconfig values.

We also have a jitterentropy warning that shows up on some hosts.
This warning is harmless, and like amba we can't turn it off in a
fragment since it is force selected by crypto (and we'd rather not
turn all crypto off). So we add it to the whitelist while investigations
continue into what is needed in the host to support this fully.

(From OE-Core rev: f5315b8c7998611da9984fd6bce2b48d6304ff6c)

Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-05 11:56:01 +01:00
Ed Bartosh 9cc4492732 image: Deploy images to IMGDEPLOYDIR
Changed deployment directory from DEPLOY_DIR_IMAGE to
IMGDEPLOYDIR to make sstate machinery to do final deployment and
generate manifest.

Renamed variable deploy_dir to deploy_dir_image in selftest code
to avoid confusion with DEPLOYDIR variable.

Updated the code of rootfs.py:Rootfs class to use IMGDEPLOYDIR variable
as it's now used as a new deployment destination.

(From OE-Core rev: 6d969bacc718e21a5246d4da9bf9639dcae29b02)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-04 00:07:28 +01:00
Francisco Pedraza 0e38688bca oeqa/selftest Adds eSDK test cases to devtool verification.
The covered functions are, install libraries headers and image generation
binary feeds.

(From OE-Core rev: 994f8a41a16d0b82a1f7dfbcbbcc1df08225b14e)

Signed-off-by: Francisco Pedraza <francisco.j.pedraza.gonzalez@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 23:45:54 +01:00
Francisco Pedraza 1da953d631 /oeqa/sdkext Adds verification for devtool on eSDK.
The covered funcions are, build make, build esdk package, build cmake
extend autotools recipe creation, kernel module,
node.js installation and recipe creation.

(From OE-Core rev: 574a5d4cf3e79815aecc4d198545119d3bbfb023)

Signed-off-by: Francisco Pedraza <francisco.j.pedraza.gonzalez@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 23:45:54 +01:00
Paul Eggleton 4b4387455c lib/oe/patch: commit with a dummy user/email when PATCHTOOL=git
When using PATCHTOOL = "git", the user of the system is not really the
committer - it's the build system itself. Thus, specify "dummy" values
for username and email instead of using the user's configured values.
Various parts of the devtool code that need to make commits have also
been updated to use the same logic.

This allows PATCHTOOL = "git" and devtool to be used on systems where
git user.name / user.email has not been set (on versions of git where
it doesn't default a value under this circumstance).

If you want to return to the old behaviour where the externally
configured user name / email are used, set the following in your
local.conf:

PATCH_GIT_USER_NAME = ""
PATCH_GIT_USER_EMAIL = ""

Fixes [YOCTO #8703].

(From OE-Core rev: 765a9017eaf77ea3204fb10afb8181629680bd82)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 23:45:52 +01:00
Paul Eggleton e5f61f85c5 oe-selftest: devtool: fix test after recent change
OE-Core commit d3057cba0b01484712fcee3c52373c143608a436 fixed handling
of wildcard bbappends, which means that this test's expectations about
the bbappend file name are no longer met. devtool finish is meant to use
wildcard bbappends so fix the test accordingly.

(From OE-Core rev: 21603566e4a2e709dcb4a940b49d870c91c822be)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 23:45:52 +01:00
Joshua Lock 1429d9d463 oeqa.selftest.liboe: add test for xattr in copytree
Add a test to ensure that oe.path.copytree() preserves extended
attributes on files.

(From OE-Core rev: 2b047b8e3218f95978e41fee13635bff9af03dd6)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:42 +01:00
Joshua Lock 92f9308e2c oe.path: preserve xattr in copytree() and copyhardlinktree()
Pass appropriate options to tar invocations in copytree() and
copyhardlinktree() to ensure that any extended attributes on the files
are preserved during the copy.

We have to drop the use cpio in "Copy-pass" mode in copyhardlinktree()
because cpio doesn't support extended attributes on files. Instead we
revert back to using cp with different patterns depending on whether
or not the directory contains dot files.

(From OE-Core rev: e591d69103a40ec4f76d1132a6039d9cb1555103)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:42 +01:00
Joshua Lock 97677c11cf oeqa.selftest: add a test for oe.path.copytree()
One motivation for the use of cpio in oe.path.copytree() was to
ensure that files with spaces in their names were copied. Add a new
unittest module to test the OE module with a test case for copytree
with a spaces in a filename.

(From OE-Core rev: a408f8310d9426db4439cf8db0cf49f9bfe90b3b)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:42 +01:00
Markus Lehtonen 7d77c02401 oeqa.buildperf: include commands log file name in results.json
(From OE-Core rev: b22a71cf3a53a33763ff02608119d2c73cbde006)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:42 +01:00
Markus Lehtonen c39db4bc45 oeqa.buildperf: include buildstats file name in results.json
No need to do lsdir magic for finding buildstats when reading results.

(From OE-Core rev: 4502f0979bf2e8698bb196345b89b170641fd43f)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:42 +01:00
Markus Lehtonen c5d1301245 oeqa.buildperf: show skipped tests in results, too
(From OE-Core rev: 4112779f9f314148b475fc4b8e33146de8be6b27)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:42 +01:00
Markus Lehtonen 81b8ccc1f6 oeqa.buildperf: convert buildstats into json format
Instead of archiving buildstats in raw text file format convert all
buildstats into one json-formatted file. Some redundant information,
i.e. 'Event:', 'utime:', 'stime:', 'cutime:' and 'cstime:' fields, are
dropped.

(From OE-Core rev: efcf74b194f2a40eb3e6359dd41386db3eb25287)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:42 +01:00
Markus Lehtonen f1fb013d48 oeqa.buildperf: measure io stat
Add data from /proc/<pid>/io to system resource measurements.

(From OE-Core rev: e69a46a77854fac1169a09e0c5b70fa4b972255a)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:42 +01:00
Markus Lehtonen 0b332039ea oeqa.buildperf: don't use Gnu time
Use Python standard library functionality instead of the time utility
for measuring elapsed (wall clock) time of commands. The time.* log
files are also ditched. However, the same detailed resource usage data,
previously found in time.* logs is now provided in results.json file.
This data is collected through the resource module of Python.

(From OE-Core rev: d5ad818dd501b18379aa3540bffa9b920d7c3bab)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:42 +01:00
Markus Lehtonen 33a38bc18a oeqa.buildperf: rename buildstats directories
Change directory name from 'buildstats-<test_name>' to just
'buildstats'. However, this patch adds the possibility to label
buildstats directory name with a postfix which makes it possible to save
multiple buildstats per test, for example.

(From OE-Core rev: 8997556040b2e7bfcfa6a75d4d97eb2e32207217)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:41 +01:00
Markus Lehtonen 6d75f39f09 oeqa.buildperf: separate output dir for each test
Store the output data of each test in an individual subdirectory instead
of storing everything in the root output directory.

(From OE-Core rev: 64ff34df96aa9a74dd4303f76ec711aa5e9d5030)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:41 +01:00
Markus Lehtonen 700ebe996a oeqa.buildperf: strip date from buildstats directory path
Archive buildstats in a directory like 'buildstats' instead of something
like 'buildstats/20160513120000'.

(From OE-Core rev: 95138cdc70bb7f9b7ab74e1d83305f009790dccc)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:41 +01:00
Markus Lehtonen e8c47a6343 oeqa.buildperf: enable json-formatted results
Automatically create a json.formatted file (results.json) in the results
directory that contains results from all tests.

(From OE-Core rev: 6df3263531a41805b2280bb999cb4a73f9f91eae)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:41 +01:00
Markus Lehtonen 44188933ce oeqa.buildperf: add 'product' to test result data
This defaults to 'oe-core' but can be defined using the
OE_BUILDPERF_PRODUCT environment variable.

(From OE-Core rev: a22cc3e04001be5d11bd85dbdceb7088cae7c735)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:41 +01:00
Markus Lehtonen 899b17413c oeqa.buildperf: treat failed measurements as errors
Now failed measurements correctly cause a test failure (recorded as an
error). There should be no need to continue the test if one step fails,
especially now that the tests don't depend on each other.

(From OE-Core rev: 446e32aadc775ca146d12173b1463f524d7fe6ef)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:41 +01:00
Markus Lehtonen 85b7b10b4a oeqa.buildperf: make tests independent
Add test set-up functionality so that the individual tests do not depend
on each other. This should make sure that a failure in one test does not
affect the results of another test. The patch also makes it reasonable
to run only a subset of the tests by using the --run-tests option.

The increase in total execution time of the full suite - caused by the
additional set-up steps - is insignificant because normally no
additional tasks need to be run. The previous test has already done all
set-up work.

(From OE-Core rev: 69b3c63e32d09ea4a41b21daacdff6bf1fc447c1)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:41 +01:00
Markus Lehtonen 6722b0412c oeqa.buildperf: fix checking of invalid results
The test status check done when writing globalres log was incorrect.

(From OE-Core rev: 3efbd49fd80d2b349a8fd44dbcd509168dbc1061)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 09:58:41 +01:00
Stephano Cetola d45a3449bc rootfs.py: allow removal of unneeded packages
Current functionality allows for the removal of certain packages
based on the read-only image feature. This patch extends this
functionality by adding the FORCE_RO_REMOVE variable, which will
remove these packages regardless of any image features.

[ YOCTO #9491 ]

(From OE-Core rev: cfb869ffd4c37c3cc8e6b3eb732c1a7b7cfc3cb0)

Signed-off-by: Stephano Cetola <stephano.cetola@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-25 23:03:48 +01:00
Markus Lehtonen 7155a9b64d oe-build-perf-test: add {git_commit_count} keyword for --commit-results-tag
Makes it possible to create easily sortable tags. Also, the default tag
format is updated to use the new keyword.

(From OE-Core rev: e3161654d75dfc3b059c519205b38b26e3ffb215)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-25 23:03:47 +01:00
Markus Lehtonen d0bac259bd oeqa.buildperf: add git commit count to result data
This number represents the number of commits since the beginning of git
history until the tested revision. This helps e.g. in ordering results.

(From OE-Core rev: b52070dd057ff5b410cd193f9be2f25bc4c506cc)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-25 23:03:47 +01:00
Markus Lehtonen caf6ad889c oe-build-perf-test: new {tag_num} keyword for --commit-results-tag
This makes it possible to create numbered tags, where the "basename" of
the tag is the same and the only difference is an (automatically)
increasing index number. This is useful if you do multiple test runs on
the same commit. For example, using:
--commit-results-tag {tester_host}/{git_commit}/{tag_num}

would give you tags something like:
myhost/decb3119dffd3fd38b800bebc1e510f9217a152e/0
myhost/decb3119dffd3fd38b800bebc1e510f9217a152e/1
...

The default tag format is updated to use this new keyword in order to
prevent unintentional tag name clashes.

(From OE-Core rev: cf2aba16338a147f81802f48d2e24a96c7133548)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-25 23:03:47 +01:00