Commit Graph

1366 Commits

Author SHA1 Message Date
California Sullivan 840ea755f2 parselogs.py: Don't clog QA with Joule errors
The Joule is very new hardware and there is ongoing kernel and firmware
work to fix these issues, which will be available in future kernel and
firmware releases. In the meantime, don't clog QA reports.

[YOCTO #10611]

(From OE-Core rev: facf9fa905100945738c13f9f79e938ed4a81030)

Signed-off-by: California Sullivan <california.l.sullivan@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-13 22:55:21 +00:00
Patrick Ohly 9eb89487ff buildstats.py: skip collecting unavailable /proc data
Some virtualized environments like Linux-VServer do not have the
entries under /proc that the new system usage sampling expected,
leading to an exception when trying to open the files.

Now the presence of these files is checked once before enabling the
corresponding data collection. When a file is missing, the
corresponding log file is not written either and pybootchart will not
draw the chart that normally displays the data.

Errors while reading or writing of data samples is intentionally still
a fatal error, because that points towards a bigger problem that
should not be ignored.

Reported-by: Andreas Oberritter <obi@opendreambox.org>
(From OE-Core rev: daeee2d6731014c33f0d1f8a3846830c099932b4)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-13 22:55:20 +00:00
Ola x Nilsson 2765231499 recipetool: selftest: Add test for recipetool plugin loading
Test that recipetool plugins are loaded in a well defined order.

(From OE-Core rev: 044de8424a454a7057906e44eb56e2134ebb17e4)

Signed-off-by: Ola x Nilsson <ola.x.nilsson@axis.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-13 22:55:20 +00:00
Ola x Nilsson 9170a88cbd devtool: selftest: add test for devtool plugin loading
Test that devtool plugins are loaded in a well defined order.

(From OE-Core rev: 0de81f0c8b29d8b442b3d099c3bec3fd345b6bfe)

Signed-off-by: Ola x Nilsson <ola.x.nilsson@axis.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-13 22:55:20 +00:00
Mariano Lopez 904b915369 oeqa/utils/metadata.py: Add metadata library
Adds functions to get metadata from the host running the tests.

[YOCTO #9954]

(From OE-Core rev: 10b05794254886e55c76f29f7778d783c550befa)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-13 22:55:19 +00:00
Mariano Lopez 9d55e9d489 oeqa/utils/commands.py: Make a copy of variables in get_bb_vars
The function get_bb_vars will remove items for the list passed
as the function argument, this will leave the caller with an
empty list and the function never says it will consume the items.

This hasn't been found before because only get_bb_var uses this
function.

(From OE-Core rev: 22b7fa24fefcc3974806d1b282c93b8c5880f6a4)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-13 22:55:19 +00:00
Juro Bystricky 90404e1f24 targetloader.py: drop test for ClassType
ClassType was removed from python3.
The code testing for ClassType kept throwing AttributeError exceptions:

    module 'types' has no attribute 'ClassType'

The exceptions prevented loading of any dynamically resolved target
controllers.

(From OE-Core rev: d62f18c39bc0ed3b0f5ac8465b393c15f2143ecf)

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12 15:16:42 +00:00
Stephano Cetola d8fbaebc48 package_manager: remove strings and migrate to direct arrays
When using subprocess call and check_output, it is better to use arrays
rather than strings when possible to avoid whitespace and quoting
problems.

[ YOCTO #9342 ]

(From OE-Core rev: b12cec9a5ef14ecb02be7feec65508cf5d65c795)

Signed-off-by: Stephano Cetola <stephano.cetola@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-08 10:31:30 +00:00
Ed Bartosh 1b32c6ed02 selftest: wic: fix test_qemu
Setting WKS_FILE variable in qemux86-64 made wic test to
use wrong wks file to produce an image and resulted in
test_qemu failure.

Used conditional assignment in qemux86-64 and explicitly
set WKS_FILE in wic testing suite to make the suite to use
wic-image-minimal.wsk. This should fix test_qemu failure.

(From OE-Core rev: 3bca4d18c2712e3b154bacfb917f0a749ebaddeb)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:38:06 +00:00
Patrick Ohly f1a527c5f6 buildstats: reduce amount of data stored for system utilization
Pre-processing /proc data during the build considerably reduces the
amount of data written to disk: 176KB instead of 4.7MB for a 20
minuted build. Parsing also becomes faster.

The disk monitor log added another 16KB in that example build. The
overall buildstat was 20MB, so the overhead for monitoring system
utilization is small enough that it can be enabled by default.

(From OE-Core rev: b17812385cd55e81066d3ceda92dffdc6e5564da)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:37:59 +00:00
Patrick Ohly 25a5536a6e buildstats: record disk space usage
Hooks into the new monitordisk.py event and records the used space for
each volume. That is probably the only relevant value when it comes to
visualizing the build and recording more would only increase disk
usage.

(From OE-Core rev: 21a5b569370f47cc02291e1d8b76fe43faa04ea6)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:37:59 +00:00
Patrick Ohly 8f475b78c9 buildstats: add system state sampling
/proc/[diskstats|meminfo|stat] get sampled and written to the same
proc_<filename>.log files as during normal bootchat logging. This will
allow rendering the CPU, disk and memory usage charts.

Right now sampling happens once a second, triggered by the heartbeat
event.That produces quite a bit of data for long builds, which will be
addressed in a separate commit by storing the data in a more compact
form.

(From OE-Core rev: 6f4e8180b5b4857eaf6caf410fd3a4a41ed85930)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:37:59 +00:00
Jair Gonzalez 5956492c20 parselogs: Whitelist GPT warnings as the device is fully functional
The warning occurs when the GPT image is not the same size than the
media into which it's being flashed, causing the backup GPT table
not being at the end of the disk. However, this is expected as the
image is created before having the information about the destination
media. The error is harmless, so it will be whitelisted.

Fixes [YOCTO 10481].

(From OE-Core rev: 5cc5cdc788308a79f8f0706e6d794c602ef427ed)

Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:37:59 +00:00
California Sullivan 23d2a7d156 parselogs.py: Whitelist iwlwifi firmware load error messages
The iwlwifi module of any given kernel has a minimum and maximum
supported firmware version. The kernel begins by attempting to load the
maximum version, and decrements until it is successful. The 4.8 kernel's
maximum supported firmware version is 24, but thus far only 22 has been
released, meaning we get errors for 24 and 23.

Filter out iwlwifi firmware load error messages, as they are not
necessarily indicative of real problems.

(From OE-Core rev: 7df570c2310efac8f9898da15deaac2b7df16655)

Signed-off-by: California Sullivan <california.l.sullivan@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:37:59 +00:00
Martin Vuille b7deb1f046 terminal.py: Pass string instead of bytes to ExecutionError to avoid exception
Based on run() in bitbake/lib/bb/process.py, ExecutionError() expects strings
not bytes. Passing bytes results in a "TypeError: Can't convert 'bytes' object
to str implicitly" exception.

Fixes Bug 10729

(From OE-Core rev: 063b63d4d324c23322ac1b6b7c7928e725d7b968)

Signed-off-by: Martin Vuille <jpmv27@yahoo.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:08 +00:00
Jose Perez Carranza b81477d329 runtime: Update test cases numbers for runtime tests
Update test case numbers on runtime tests to do match
with templates defined on Testopia for 2.3 release

(From OE-Core rev: d9df762b4c62b74f6d3a1521642ea86c26793a22)

Signed-off-by: Jose Perez Carranza <jose.perez.carranza@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:08 +00:00
Michael Wood cfdc5b853e oe-selftest: toaster Remove redundant Toaster test
This test has been ported to be run as part of Toaster's own tests.

(From OE-Core rev: f6366781a34dbdb8ec2d73f4fb36359bc15c4f42)

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:06 +00:00
California Sullivan f3c9e66e8d parselogs: Whitelist NUC6 firmware load error message on genericx86-64
This was already whitelisted, but the 4.8 kernel changed the error
message, causing it to get caught by parselogs again.

Fixes [YOCTO #10494].

(From OE-Core rev: e1bad14231115f3b1a2bf844ef5b2022c648b55d)

Signed-off-by: California Sullivan <california.l.sullivan@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:14 +00:00
Joshua Lock 333d300fba lib/oe/lsb: better handle missing fields
Some rolling release distros, such as Arch Linux, don't include a
VERSION_ID field in their os-release file.

Change release_dict_osr() to better handle this optional field
being absent.

Further improve the resilience of the release_dict_*() methods by
always returning a dict and using dict.get() in distro_identifier()
to supply a default, empty string, value when then key is missing.

(From OE-Core rev: e36066dcc3b56cac1c695370ea178b566c0ebfd6)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:14 +00:00
Paul Eggleton 6e182aaa76 devtool: modify: support recipes with only local files as source
The hello-mod recipe is unusual in that it has only local files in
SRC_URI and builds these out of ${WORKDIR}. When you use devtool modify
on it, devtool puts all of those files in an "oe-local-files"
subdirectory of the source tree, which is not ${S} (or ${B}) any more
and thus building the recipe afterwards fails. It's a bit of a hack, but
symlink the files in oe-local-files into the source tree (and commit the
symlinks with an ignored commit so that the repo is clean) to work
around the problem. We only do this at time of extraction, so any files
added to or removed from oe-local-files after that won't be handled, but
I think there's a limit to how far we should go to support these kinds
of recipes - ultimately they are anomalies.

I initially tried a hacky workaround where I set effectively set B =
"${WORKDIR}" and that allowed it to build, but other things such as the
LIC_FILES_CHKSUM checks still broke because they expected to find files
in ${S}. Another hack where I set the sourcetree to point to the
oe-local-files subdirectory works for hello-mod but not for makedevs
since whilst that is similar, unlike hello-mod it does in fact have
files in the source tree (since it has a patch that adds COPYING) and
thus the same issue occurred.

Also tweak one of the tests that tries devtool modify / update-recipe on
the makedevs recipe to try building it since that would have caught this
issue.

Fixes [YOCTO #10616].

(From OE-Core rev: 857c06d6a1d161bf5a01311d07758bd4241929a3)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:14 +00:00
Stephano Cetola 93e8db2224 devshell: list commands when throwing NoSupportedTerminals
When attempting to run devshell, if no terminal is available, the
error being thrown was not very specific. This adds a list of
commands that failed, informing the user of what they can install to
fix the error.

[ YOCTO #10472]

(From OE-Core rev: c077f4aab2fc956408d4ad45c4e2e2ea6e480624)

Signed-off-by: Stephano Cetola <stephano.cetola@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:13 +00:00
Paul Eggleton 39fba4d70f oe-selftest: devtool: test that updating a file with subdir= works
If you have a file:// entry in SRC_URI with a subdir= parameter that
makes it extract into the source tree, then when you update that file in
oe-local-files and run devtool update-recipe then you want the original
file to be updated. This was made to work by OE-Core commit
9069fef5dad5a873c8a8f720f7bcbc7625556309 together with
31f1bbad248c36a8c86dde4ff57ce42efc664082, however until now there was no
oe-selftest test to verify it.

Note that in order to succeed this test also requires the fix
"lib/oe/recipeutils: ignore archives by default in
get_recipe_local_files()" since the test recipe uses a local tarball.

(From OE-Core rev: 936eba3e1059d1dcd5e58c1ce76870fff7b11b3c)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:13 +00:00
Paul Eggleton e3193b76e0 lib/oe/recipeutils: ignore archives by default in get_recipe_local_files()
By default, have get_recipe_local_files() not return any archive
files. This prevents a local tarball from being erroneously removed
from SRC_URI if you run "devtool modify" on a recipe followed by
"devtool update-recipe". It doesn't actually help you to directly
update the contents of such tarballs, but at least now it won't break
the recipe.

(From OE-Core rev: e9c418d4704c1bed4c5880e176e5288485f4f5a6)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:13 +00:00
Paul Eggleton 55a157f4e6 devtool: update-recipe: fix handling of compressed local patches
It is possible to use gzip or bzip2 to compress patches and still refer
to them in compressed form in the SRC_URI value within a recipe. If you
run "devtool modify" on such a recipe, make changes to the commit for
the patch and then run devtool update-recipe, we need to correctly
associate the commit back to the compressed patch file and re-compress
the patch, neither of which we were doing previously.

Additionally, add an oe-selftest test to ensure this doesn't regress in
future.

Fixes [YOCTO #8278].

(From OE-Core rev: e47d21624dfec6f71742b837e91da553f18a28c5)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:13 +00:00
Paul Eggleton d316363b7b lib/oe/patch: fix handling of patches with no header
If a patch applied by a recipe has no header and we turn the recipe's
source into a git tree (when PATCHTOOL = "git" or when using devtool
extract / modify / upgrade), the commit message ends up consisting only
of the original filename marker ("%% original patch: filename.patch").
When we come to do turn the commits back into a set of patches in
extractPatches(), this first line ends up in the "Subject: " part of
the file, but we were ignoring it because the line didn't start with the
marker text. The end result was we weren't able to get the original
patch name. Strip off any "Subject [PATCH x/y]" part before looking for
the marker text to fix.

This caused "devtool modify openssl" followed by "devtool update-recipe
openssl" (without any changes in-between) to remove version-script.patch
because that patch has no header and we weren't able to determine the
original filename.

(From OE-Core rev: d9971f5dc8eb7de551fd6f5e058fd24770ef5d78)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:12 +00:00
Paul Eggleton e5a391795a oe-selftest: devtool: test update-recipe with only local files
Add a test to ensure devtool update-recipe works properly on recipes
that contain only local files (since the other tests we have didn't test
that).

Relates to [YOCTO #10563].

(From OE-Core rev: bdc844b3f0c3fbddcd3523095899a5bd29797704)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:12 +00:00
Linus Wallgren b91e47fccb lib/oe/package_manager: .deb pre/postinst args
The debian policy manual and MaintainerScripts wiki page states that the
postinst script is supposed to be called with the `configure` argument
at first install, likewise the preinst script is supposed to be called
with the `install` argument on first install.

https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html
https://wiki.debian.org/MaintainerScripts

(From OE-Core rev: 3d9c3aae54589794ce3484fa1b21d1af2bd32661)

Signed-off-by: Linus Wallgren <linus.wallgren@scypho.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:11 +00:00
Joshua Lock 81afedc905 lib/oe/lsb: attempt to ensure consistent distro id regardless of source
The LSB Distributor ID and os-release NAME differ for most of the
distributions tested by the Yocto Project (CentOS, Debian, Fedora,
openSUSE and Ubuntu) however for all but openSUSE the os-release ID
matches the LSB Distributor ID when both are lowered before
comparison.

Therefore, in order to improve the consistency of identification of
a distribution, switch to using the os-release ID and converting
the ID value to lowercase.

Table showing comparison of LSB Distributor ID to os-release fields NAME
and ID for current Yocto Project supported host distributions:

Distribution | Version | Distributor ID   | NAME             | ID       |
-------------------------------------------------------------------------
CentOS       | 7       | CentOS           | CentOS Linux     | centos   |
Debian       | 8       | Debian           | Debian GNU/Linux | debian   |
Fedora       | 23      | Fedora           | Fedora           | fedora   |
Fedora       | 24      | Fedora           | Fedora           | fedora   |
openSUSE     | 13.2    | openSUSE project | openSUSE         | opensuse |
openSUSE     | 42.1    | SUSE LINUX       | openSUSE Leap    | opensuse |
Ubuntu       | 14.04   | Ubuntu           | Ubuntu           | ubuntu   |
Ubuntu       | 16.04   | Ubuntu           | Ubuntu           | ubuntu   |

[YOCTO #10591]

(From OE-Core rev: 8689e5618d45c2119134ea64754430c06a93ea09)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-15 15:19:54 +00:00
Joshua Lock 42441ea481 lib/oe/lsb: prefer /etc/os-release for distribution data
os-release(5) is an increasingly standard source of operating system
identification and more likely to be present on modern OS deployments, i.e.
many container variants of common distros include os-release and not the
lsb_release tool.

Therefore we should favour parsing /etc/os-release in distro_identifier(),
try lsb_release when that fails and finally fall back on various distro
specific sources of OS identification.

(From OE-Core rev: fc4eddecddec68d03a985086fa32db40ad0c7bfc)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-15 15:19:54 +00:00
Joshua Lock 545f5f96d9 lib/oe/lsb: make the release dict keys consistent regardless of source
Rather than have the distro_identifier method look for different keys in
the dict depending on the source ensure that each function for retrieving
release data uses the same key names in the returned dict.

(From OE-Core rev: 2ddd6ddaf0c5ba14ae83347eba877ac9ef179c76)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-15 15:19:54 +00:00
Ross Burton 025f917cec lib/oe/qa: handle binaries with segments outside the first 4kb
The ELF parser was assuming that the segment tables are in the first 4kb of the
binary.  Whilst this generally appears to be the case, there have been instances
where the segment table is elsewhere (offset 2MB, in this sample I have).  Solve
this problem by mmap()ing the file instead.

Also clean up the code a little whilst chasing the problem.

(From OE-Core rev: a66660aa5bb709547ce0b65a4563e4217c3c3d9f)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-15 15:19:53 +00:00
Joshua Lock 5193dfdb1f lib/oe/path: remove duplicate import
There's no need to import glob inside copyhardlinktree() as it's
already imported for the entire path module.

(From OE-Core rev: 42dc4695da136a15bebb7525b1da5c2722b10a28)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-15 15:19:53 +00:00
Ross Burton 4e6c5d8769 distro_check: MeeGo is long dead, compare against Clear Linux instead
Instead of checking against a file that represents a distribution that hasn't
existed for years, fetch package names for Clear Linux instead.

[ YOCTO #10601 ]

(From OE-Core rev: 006c4db0974c42ff0f6950dd24e61c008f801679)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-15 15:19:53 +00:00
Ross Burton b654f5175d Revert "oeqa/selftest/kernel.py: Add new file destined for kernel related tests"
breaking on selftest

NOTE: recipe core-image-minimal-1.0-r0: task do_rootfs: Started
ERROR: core-image-minimal-1.0-r0 do_rootfs: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:license_create_manifest(d)
     0003:
File: '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/classes/license.bbclass', lineno: 48, function: license_create_manifest
     0044:    pkg_dic = {}
     0045:    for pkg in sorted(image_list_installed_packages(d)):
     0046:        pkg_info = os.path.join(d.getVar('PKGDATA_DIR', True),
     0047:                                'runtime-reverse', pkg)
 *** 0048:        pkg_name = os.path.basename(os.readlink(pkg_info))
     0049:
     0050:        pkg_dic[pkg_name] = oe.packagedata.read_pkgdatafile(pkg_info)
     0051:        if not "LICENSE" in pkg_dic[pkg_name].keys():
     0052:            pkg_lic_name = "LICENSE_" + pkg_name
Exception: FileNotFoundError: [Errno 2] No such file or directory: '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/sysroots/qemux86-64/pkgdata/runtime-reverse/kernel-4.8.3-yocto-standard'

This reverts commit c3d2df883a.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-15 15:19:52 +00:00
Paul Eggleton 43e652f3d1 devtool: add "rename" subcommand
When you run devtool add on a source tree we attempt to figure out the
correct name and version for the recipe. However, despite our best
efforts, sometimes the name and/or version we come up with isn't
correct, and the only way to remedy that up until now was to reset the
recipe, delete the source tree and start again, specifying the name this
time. To avoid this slightly painful procedure, add a "rename"
subcommand that lets you rename the recipe and/or change the version.

(From OE-Core rev: 9303d8055c45a0f6af295d70a6f6a8b9d8d8a7c9)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-07 11:04:22 +00:00
Paul Eggleton 7aebaa4204 oe-selftest: devtool: fix error message in _test_recipe_contents()
If a variable is being set in the recipe when we've explicitly passed
None as the value to _test_recipe_contents() indicating that it
shouldn't be set at all, then we should be printing out the variable
name in the assertion message but it seems like I forgot to do a
substitution. Also include the value for informational purposes.

(From OE-Core rev: 0dafcb158003fb13f82c266f607d9967fca321db)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-07 11:04:22 +00:00
Paul Eggleton e0b62c38ac recipetool: create: separate LICENSE items with & by default
recipetool sets the LICENSE value based on licenses detected from the
source tree. If there are multiple licenses then they were being
separated by spaces, but this isn't actually legal formatting and if
you're using "devtool add" you get a warning printed when devtool
parses the recipe internally.

Earlier I had made a conscious decision to do it this way since it's up
to the user to figure out whether the multiple licenses should all apply
(in which case they'd be separated with &) or if there is a choice of
license (in which case | is the correct separator). However, I've come
to the conclusion that we can just default to & and then the ugly
warning goes away, and it's the safest alternative of the two (and most
likely to be correct, since it's more common to have a codebase which is
made up of code with different licenses, i.e. all of them apply to the
combined work).

I've tweaked the comment that we add to the recipe to explicitly state
that we've used & and that the user needs to change that if that's not
accurate.

Fixes [YOCTO #10413].

(From OE-Core rev: ecac6aee8cf3313350b58c21012bcd67cfb915e4)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-07 11:04:22 +00:00
Ross Burton 1655d55764 distro_check: partial rewrite to make it work again
This library suffered as part of the Python 2 to Python 3 migration and stopped
working entirely.

Fix all the migration problems such as files being treated as strings but opened
in binary mode, insufficient use of with on files, and so on.

Rewrite large amounts to be Pythonic instead of C-in-Python.

Update OpenSuse and Fedora URLs.

Fedora now splits the archive alphabetically so handle that.

[ YOCTO #10562 ]

(From OE-Core rev: 58de12eaaac9c60bb8fc84a3a965ef86d2a39ae0)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-06 23:35:36 +00:00
Paul Eggleton 385f108377 lib/oe/recipeutils: print just filename in bbappend_recipe() if in temp dir
If you use devtool update-recipe with the --append option, and a "local"
(in oe-local-files) has been modified we copy it into the specified
destination layer. With the way the devtool update-recipe code works now
the source is always a temp directory, and printing paths from within
that is just confusing, so if the path starts with the temp directory
then just print the file name alone.

(From OE-Core rev: 61475f0267d40c618ebf36023d0b6414a25975cb)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-06 23:35:36 +00:00
Samuli Piippo b18901d9bc package_manager.py: correctly remove all dependent packages
Do not use --force-depends when trying to remove all dependent packages,
as it removes only the selected package and not the dependent packages.

(From OE-Core rev: a82e8725902086dab785a0b14305927dae1e4e8d)

Signed-off-by: Samuli Piippo <samuli.piippo@qt.io>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-06 23:35:33 +00:00
Robert Yang d31d1ad4e5 oe/copy_buildsystem.py: dereference symlink
When there is a relative symlink in the layer, for example:
symA -> ../out/of/layer/file

symA will be invalid fater copied, it would be invalid from build time
if it points to a relative path, and would be invalid after extracted
the sdk if it points to a absolute py. Dereference symlink when copy
will fix the problem.

Use tar rather than shutil.copytree() to copy is because:
1) shutil.copytree(symlinks=Fasle) has bugs when dereference symlinks:
   https://bugs.python.org/issue21697
   And Ubunutu 1404 doesn't upgrade python3 to fix the problem.

2) shutil.copytree(symlinks=False) raises errors when there is a invalid
   symlink, and tar just prints a warning, tar is preferred here since
   the real world is unpredicatable

3) tar is faster than shutil.copytree() as said by oe.path.copytree()

So use tar to copy.

(From OE-Core rev: f4d70bb0882eec4fb46cd942f2796fad57c72982)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-06 23:35:32 +00:00
Costin Constantin c3d2df883a oeqa/selftest/kernel.py: Add new file destined for kernel related tests
[YP#7202]:  Test for linux-dummy
The new kernel.py file is intended for kernel related test cases.
The test for linux-dummy will ensure it is in good shape and can
be used as a kernel replacement at build time. To do this, the
test will first clean sstate for linux-dummy target, ensuring no
file is present in the stamps directory. After, core-image-minimal
is built, ensuring linux-dummy can be used as a kernel substitute.

(From OE-Core rev: 98c6ebf1e05158c689e01b785d32757847cdb10c)

Signed-off-by: Costin Constantin <costin.c.constantin@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-01 10:05:46 +00:00
Davis, Michael e66f5eabed terminal.py: Add compatiblity for konsole 16.08.1
Konsole has dropped support for the nofork flag.  It has been replaced with the seperate flag.

(From OE-Core rev: f0b193b63d4c468c3aa58e15ef5a991e04b9b9a2)

Signed-off-by: Michael Davis <michael.davis@essvote.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-28 16:15:18 +01:00
Joshua Lock 751cb2c894 Remove RM_OLD_IMAGE, it's no longer useful
Since the move to put image deployment under sstate control in
d54339d4b1a7e884de636f6325ca60409ebd95ff old images are automatically
removed before a new image is deployed (the default behaviour of the
sstate logic).

RM_OLD_IMAGE is therefore no longer required to provide this
behaviour, remove the variable and its users.

(From OE-Core rev: 93631befe8b962bf99524746b49f4ebca336175c)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-15 10:01:42 +01:00
Ross Burton 84198174bf lib/oe/qa: add ELF machine to string function
Add a function (and test suite) to turn the ELF machine field (e_machine) into a
string, so we can tell the user "x86-64" instead of 0x3E.

(From OE-Core rev: 72336003741fb16a7ecdd6b753eae56310413ff7)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-11 22:19:21 +01:00
Richard Purdie 78c01995e3 oeqa/sshcontrol: Handle interrupted system call error
Deal with an interrupted system call gracefully:

|   File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-qa-systemd/build/meta/lib/oeqa/utils/sshcontrol.py", line 55, in _run
|     if select.select([self.process.stdout], [], [], 5)[0] != []:
| InterruptedError: [Errno 4] Interrupted system call

(From OE-Core rev: 556125e4004fb7ac5169b59f51dc151f18c1806a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-07 16:43:58 +01:00
Ed Bartosh 09a9de45e1 wic: selftest: add test for sdimage-bootpart
Test creation of sdimage-bootpart image

(From OE-Core rev: 8b18568591f66aa9770798bc61123898de92b8a9)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-07 16:43:58 +01:00
Ed Bartosh bed4a13440 wic: selftest: add test for systemd-bootdisk
Test creation of systemd-bootdisk image.

(From OE-Core rev: b8eef88536b944ea35664b9a6a9496fb0a3f7988)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-07 16:43:58 +01:00
Peter Kjellerstedt d067b0e1e4 package_manager.py: Allow multiple regexps in PACKAGE_EXCLUDE_COMPLEMENTARY
The PACKAGE_EXCLUDE_COMPLEMENTARY variable can currently only contain
one regular expression. This makes it hard to add to it from different
configuration files and recipes.

Allowing it to contain multiple, whitespace separated regular
expressions should be backwards compatible as it is assumed that
whitespace is not used in package names and thus is not used in any
existing instances of the variable.

After this change, the following three examples should be equivalent:

  PACKAGE_EXCLUDE_COMPLEMENTARY = "foo|bar"

  PACKAGE_EXCLUDE_COMPLEMENTARY = "foo bar"

  PACKAGE_EXCLUDE_COMPLEMENTARY = "foo"
  PACKAGE_EXCLUDE_COMPLEMENTARY += "bar"

(From OE-Core rev: a5f7e98a94e96d40b1276c85249619aa8d7be847)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-06 07:51:00 +01:00
Peter Kjellerstedt 2334201960 package_manager.py: Allow a leading - in PACKAGE_EXCLUDE_COMPLEMENTARY
This allows a regular expression specified in
PACKAGE_EXCLUDE_COMPLEMENTARY to have a leading dash. Without this,
the dash was treated by oe-pkgdata-util as the beginning of a command
line argument. E.g., if PACKAGE_EXCLUDE_COMPLEMENTARY = "-foo$", it
resulted in an error like:

  ERROR: <imagename>-1.0-r0 do_populate_sdk: Could not compute
  complementary packages list. Command '<topdir>/scripts/oe-pkgdata-util -p
  <builddir>/tmp/sysroots/<machine>/pkgdata glob
  <workdir>/installed_pkgs.txt *-dev *-dbg -x -foo$' returned 2:
  ERROR: argument -x/--exclude: expected one argument
  usage: oe-pkgdata-util glob [-h] [-x EXCLUDE] pkglistfile glob [glob ...]

(From OE-Core rev: ac4ca41d3a27356d46c0c39053e74d3519b24c44)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-06 07:51:00 +01:00