Commit Graph

1925 Commits

Author SHA1 Message Date
Ed Bartosh 0c57dd96c8 wic: replace print statements with print function
Print statements have been replaced with print function in
Python 3. Replaced them in wic code to be able to run it
under both Python 2 and Python 3.

[YOCTO #9412]

(From OE-Core rev: ee6979a19c77931c3cf6368e695e370d46192fef)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-14 23:05:13 +01:00
Patrick Ohly 4dadbbdd46 combo-layer: avoid too long command lines in update with history
As suspected, invoking "git archive" with all intended files as
parameters can run into command line length limitations. Splitting up
the parameters into multiple invocations (xargs-style) works and was
tested after encountering the situation in practice.

(From OE-Core rev: 1cb484ab99eabb5c24792757ab09d7f170f2e614)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-14 23:05:13 +01:00
Patrick Ohly 659ed8718b combo-layer: fix default "update" mode
When the "history" option is not set in the combo-layer.conf, the
intended default was to use the traditional method. Passing "True" as
default when querying the config was unintentional.

Also remove some left-over debugging code.

(From OE-Core rev: d0304acb05b926b08805d8652e12eaf19bf53ad6)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-14 23:05:12 +01:00
Randy Witt 832029356f devtool: Fix build-sdk when pn doesn't match filename
If an image with the filename foo.bb could be built using the name "bar"
instead, then build-sdk would fail to create the derivative sdk.

This was because the code assumed that the file name matched the target,
which is not necessarily the case.

(From OE-Core rev: d58a326b6960be14b8a049253559aec9582b7d0d)

Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-11 10:33:43 +01:00
Patrick Ohly d53ed05390 combo-layer: implement "update with history"
The core idea is that all commits get imported, including merge
commits, and joined into one big merge commit that imports the changes
from the individual components into the main branch of the combined
repository.

This is done by copying the files in each commit and removing deleted
ones, instead of trying to patch the combined repository.

The advantages of doing updates in this mode are:
- works for arbitrary upstream repos, not just those which
  support conversion into a linear set of patches
- listing history shows that commits where developed
  independently in the different components, instead of
  artificially showing them as if they had been developed
  one after the after (component "aaaa" before "bbbb", then "ccc", ...)
- bisecting becomes easier: when upstream repos only ensure consistency
  when merging into their "master" branches, then those merge
  commits are good candidates for test builds also in the combined
  repo
- more data mining can be done, for example showing who merged a commit
  and when

Selecting a subset of the files is supported, albeit with a slight
different semantic for wild card matching compared to other code paths
(/ is matched by * and ?). Empty commits get skipped because typically
they are a result of filtering (but that is not checked, so
intentionally empty commits also get skipped).

Other code paths are intentionally left unchanged, to avoid
regressions. However, the downside is that some opportunities for
refactoring (in particular regarding file filtering) were ignored.

(From OE-Core rev: 660f76b6fb0fb95738a2c8f50e0a99ffa5831c64)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-11 10:33:41 +01:00
Patrick Ohly 107c35e87e combo-layer: runcmd() enhancements
Allow setting the environment. Due to a subprocess quirk, it must
always be set explicitly (reuses the one from the previous call if not
set, instead of falling back to os.environ).

Embedding nul characters will be useful for parsing git output more
reliably; support dumping such output a bit better.

(From OE-Core rev: 0af4fadafce690fc8357196cf7247bd222c08d10)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-11 10:33:41 +01:00
Patrick Ohly aa4de3c8c5 combo-layer: dummy "update with history"
When setting "history = True" in combo-layer.conf consistently for the
components involved in an update or using "update" together with the
"--history" command line flag, a new mode for updating will be used
that does not rely on exporting/importing patches.

A config setting is used because it should be used consistently by
everyone using the same config, without having to remember to use an
additional command line parameter.

There are no real global settings, so the setting is checked
separately for each component although the setting has to be set
consistently. This restriction could be removed later.

In practice, putting "history" into the "[DEFAULT]" section is the
easiest approach for configuring it.

The actual code changes split up action_update and the
combo-layer.conf handling in preparation for this new mode, without
implementing the mode itself.

(From OE-Core rev: c9dab31f5f6dc225f5c2c2ca3ec9aeab2ff655d5)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-11 10:33:41 +01:00
Leonardo Sandoval 912e372bd3 scripts/lib/bsp/kernel.py: force patching when branch is machine branch is re-use
When a branch is re-used, the kernel tools turns off any patch pushing unless
'mark patching' is explicitly set.

[YOCTO #9120]

(From meta-yocto rev: 427f5473722e15e288cbce251a9ce18989c23548)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-06 10:32:24 +01:00
Christopher Larson 6e4b817776 ddimage: if 'pv' is installed, use it
This gives us a progress bar for the image write, which is quite helpful. See
https://www.ivarch.com/programs/pv.shtml.

(From OE-Core rev: 5b20cb2db929daaf0bf81c05368174e9c364ffab)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-06 10:31:15 +01:00
Paul Eggleton 8f275ff0d8 recipetool: create: fix falling back to declared license for npm packages
Fix two problems falling back to the "license" field from package.json
when no license file is present:
1) The function that was supposed to return the license field value was
   always explicitly returning None, and this was never noticed (because
   the test cases never exercised the fallback as they provided license
   files for each module).
2) Fix the main package not falling back because it had a default of an
   empty list, which evaluates to '' instead of 'Unknown'.

(From OE-Core rev: 59381a9450949ce6b4b03adb717e950b999830f3)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-06 10:31:14 +01:00
Christopher Larson a78e4ade1f scripts/lib/argparse_oe: also change 'positional arguments' to 'arguments'
This aligns with our existing 'optional arguments' to 'options' change, and
seems more intuitive for users.

(From OE-Core rev: 8a1cd471210e5fb77952f28172084bf6a4fb73e8)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-06 10:31:13 +01:00
Christopher Larson 09cbec8947 scripts/lib/argparse_oe: simplify options title change
There's no need to iterate over the action groups here, as self._optionals and
self._positionals are available.

(From OE-Core rev: 408694f4320f3cb52a391e5b927fb8c8ba16c1d2)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-06 10:31:13 +01:00
Christopher Larson 3e79d54523 scripts/lib/argparse_oe: show subparser help for unrecognized args
As an example, `recipetool create foo bar baz` shows `recipetool: error:
unrecognized arguments: bar baz` and then displays the main help, not the help
for the create command. Fix by saving the subparser name and using it in
parse_args() to look up the subparser.

(From OE-Core rev: 7fdaaedf4c63c8d019f03f84e22f9b838ef19aa6)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-06 10:31:13 +01:00
Christopher Larson 55c760bda3 scripts/lib/argparse_oe: show self.prog in the error message
This aligns our subclassed error() with that in the original class, using
_print_message and self.prog. Also add a docstring based on the original.

(From OE-Core rev: cf0c5175136966eefde8c0d9aa0679e85779f713)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-06 10:31:13 +01:00
Ed Bartosh 7669219652 wic: use sparse_copy to preserve sparseness
Used sparse_copy API in favor of dd/cp in rawcopy plugin to
preserve sparseness of the copied raw content.

[YOCTO #9099]

(From OE-Core rev: 04eca59068a79ae6a9969be495c4cdf0c5c3e466)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-06 10:31:13 +01:00
Ed Bartosh db9557cc46 wic: use sparse_copy to copy partitions
Copied partition images into final partitioned image using
sparse_copy API. This method preserves sparseness of the
final image. It also makes wic much faster, as unmapped
blocks of the partition images are not copied.

[YOCTO #9099]

(From OE-Core rev: 7f21427aca5df81d8881027fd98f71b821cf31d7)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-06 10:31:13 +01:00
Ed Bartosh 4b971568e1 wic: add sparse_copy API
In order to make wic images sparse sparse_copy function has been
copied from meta-ostro:
https://github.com/kad/meta-ostro/blob/master/meta-ostro/lib/image-dsk.py
This function uses filemap APIs to copy source sparse file into
destination file preserving sparseness.

The function has been modified to satisfy wic requirements:
parameter 'skip' has been added.

[YOCTO #9099]

(From OE-Core rev: bfde62bdc03152a4d3d383512479b974fa867f94)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-06 10:31:13 +01:00
Ed Bartosh a468d4bc57 wic: add FIEMAP and SEEK_HOLE / SEEK_DATA APIs
In order to make wic images sparse set of APIs has been copied
from bmap-tools project.

filemap.py module is taken from bmap-tools project:
https://github.com/01org/bmap-tools/blob/master/bmaptools/Filemap.py
It implements two ways of get information about file block: FIEMAP
ioctl and the 'SEEK_HOLE / SEEK_DATA' features of the file seek
syscall.

Note that this module will be removed as soon as bmaptool utility
supports copying sparse source file into destination file (this is
already agreed with the maintainer of bmap-tools project).

[YOCTO #9099]

(From OE-Core rev: 182639ddc9cda85c896a54c1c64fd1fb145071a1)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-06 10:31:13 +01:00
Ed Bartosh f8959d48e8 wic: moved DiskImage to direct.py
Moved DiskImage class from utils/fs_related.py to
imager/direct.py as it's only used there.

Removed fs_related module as it doesn't contain anything
except of DiskImage.

(From OE-Core rev: b3cc471790784c28f9362fcd6fc6a81c4316754c)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-06 10:31:13 +01:00
Ed Bartosh f3180152a4 wic: get rid of fs_related.makedirs
Removed fs_related.makedirs as is not used anywhere. The name is
easy to confuse with os.makedirs.

(From OE-Core rev: 796b114863ef20fbc89da45dbe6780abe1256f5e)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-06 10:31:12 +01:00
Ed Bartosh 4daf903fb0 wic: get rid of inheritance Disk->DiskImage
There is no need in this inheritance as DiskImage class
is used only in one module and no other classes are inherited.

(From OE-Core rev: 5af1d9bedc2c961eb91faf80251f24c3df754d76)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-06 10:31:12 +01:00
Ed Bartosh d03cbac796 wic: use truncate utility to create sparse files
Used truncate instead of dd to create wic images for the
following reasons:
 - dd doesn't preserve sparseness
 - truncate syntax is much more clear
 - dd requires additional calculations of the image size
   in blocks
 - the way dd was used in the code is not always correct.
   In some cases it was writing one block to the file which makes
   it not 100% sparse.

[YOCTO #9099]

(From OE-Core rev: d2d0d18dfd3922411d856b98ab6ba5d64c9c1c9f)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-06 10:31:12 +01:00
Ed Bartosh 2daea56eb8 wic: add help for --system-id option
Added explanation of --system-id option to the output of
wic help kickstart.

[YOCTO #9096]

(From OE-Core rev: 1a304afea4ad7be12ed5f0fcb397a538345a6b63)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-29 07:58:44 +01:00
Ed Bartosh 5bc1e42b03 wic: set partition system id
Used sfdisk to set partition system id if --system-id parameter
is used for a partition in wks file.

[YOCTO #9096]

(From OE-Core rev: a1f7f7e61fd20fb6319825648930f7b6aa0e0cee)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-29 07:58:44 +01:00
Ed Bartosh 514a87877e wic: add system_id argument to Image.add_partition
Added new argument to add_partition call to pass partition
system id down the stack.

[YOCTO #9096]

(From OE-Core rev: f2733df697192c0010c17b7bbb02f8679cb8f313)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-29 07:58:44 +01:00
Ed Bartosh 8d747c8b43 wic: add system_id attribute to Partition
Added Partition.system_id attribute and initialized it
from parse result of wks option --system-id. It will be
used by the wic code below the call stack to set partition
system id.

[YOCTO #9096]

(From OE-Core rev: 4f195a5b7574ebff8fbdb3045daa71f173f97a30)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-29 07:58:44 +01:00
Ed Bartosh e6a523286e wic: add sfdisk to the list of utilities
Added sfdisk -> util-linux pair to the dictionary
executable -> recipe as sfdisk is going to be used by wic
to set partition system id.

[YOCTO #9096]

(From OE-Core rev: 398aafa185acbc7239505f7107735e93a502f6d2)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-29 07:58:44 +01:00
Ed Bartosh a48630b12a wic: add --system-id wks option
Added new option --system-id to wks parser. The option
will be used to set partition system id.

[YOCTO #9096]

(From OE-Core rev: b9c56b1c95cd1d0fd809d257e0cd05a50c481bed)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-29 07:58:44 +01:00
Ioan-Adrian Ratiu 4ed22edac8 wic: isoimage-isohybrid: fix splash file paths
os.path.join discards the cr_workdir var contents if the path of the
second arguments is absolute.

(From OE-Core rev: dba099d77dcc66b239523a55f3ed26784f9a662a)

Signed-off-by: Ioan-Adrian Ratiu <adrian.ratiu@ni.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-29 07:58:44 +01:00
Ioan-Adrian Ratiu 222c5c65b1 wic: isoimage-isohybrid: add grubefi configfile support
The latest wic kickstart refactoring introduced a bootloader option
"--configfile" which lets wks' specify a custom grub.cfg for use
while booting. This is very useful for creating stuff like boot menus.

This change lets isoimage-isohybrid use --configfile; if this option is
not specified in a wks, it generates a default cfg as before.

(From OE-Core rev: bf673a769514b13558ad9c785ae4da3a5adfd1e0)

Signed-off-by: Ioan-Adrian Ratiu <adrian.ratiu@ni.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-29 07:58:44 +01:00
Christopher Larson 592e5a94e6 recipetool.newappend: fix syntax error for 'not path_ok' error
(From OE-Core rev: bdb5a6a5b3c31ed44bed8321f5febb6a09dfb9f2)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-29 07:41:43 +01:00
Paul Eggleton 519600c607 devtool: sdk-update: fix handling of UNINATIVE_CHECKSUM changes
If UNINATIVE_CHECKSUM changes over an SDK update, bitbake within the
extensible SDK will be broken because it will see that the matching
uninative tarball doesn't exist and if there is a default value of
UNINATIVE_URL it will attempt to download the file and will then fail
because the checksums don't match up; alternatively if no UNINATIVE_URL
is set then it'll also fail with an error about misconfiguration. To fix
this, add some logic to devtool sdk-update to download the matching
uninative tarball(s) for the checksum(s) in the newly fetched SDK
configuration.

Fixes [YOCTO #9301].

(From OE-Core rev: 14ff58ad98a5afac08db77068d80f152d8875766)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-19 21:11:24 +01:00
Peter Kjellerstedt 8953d83028 oe-selftest: Correct the usage examples
(From OE-Core rev: 7f5d150e4f9270e586c8a279ee64561c54689ffc)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-18 16:28:22 +01:00
Stephano Cetola dee47ad159 devtool: sdk-update: reset git metadata on update
Replace git pull with fetch and reset to avoid the merge logic in the
event that the layers repo in the published SDK we're updating to isn't
fast-forward merge from the local repo.

Also add gitignore and committer info during publish to avoid errors and
to be sure that the first commit has a dummy user in it.

[ YOCTO #9368 ]

(From OE-Core rev: 4657bc9d165e51981e034e73e7b92552e873eef7)

Signed-off-by: Stephano Cetola <stephano.cetola@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-18 16:28:22 +01:00
Paul Eggleton 60a8719e6e devtool: upgrade: handle recipes where source is not first entry in SRC_URI
It is unusual but not impossible to find recipes whose first entry is
not the main source URL but instead some patch or other local file, for
example python-cryptography in meta-python (which sets SRC_URI before
inheriting pypi). There's nothing inherently wrong with this, and we
shouldn't assume that the first entry is the main source URL, so just
take the first non-local entry instead.

(From OE-Core rev: c868198c1f6006789707b497c2ae34d7cc5e706f)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-15 06:59:43 +01:00
Robert Yang 1db3dc8803 runqemu: let ramfs equal to cpio.gz
For example, support both:
$ runqemu qemux86-64 ramfs
$ runqemu qemux86-64 cpio.gz (new)

(From OE-Core rev: 6529264776701d4f5a1e4a8336ac2e01a6ddea85)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-14 10:58:34 +01:00
Ed Bartosh f7352ca2c4 wic: fix bug in handling fsoptions
Partitions specifying --fsoptions were silently skipped by wic
due to the old bug introduced when removing code related to
subvolume handling:

- if mountpoint == "/" or not fsopts or fsopts.find("subvol=") == -1
+ if mountpoint == "/" or not fsopts:

[YOCTO #9396]

(From OE-Core rev: be7ff1741e8ab5f2724b3f64da1bed8b0d3dcb7c)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-14 10:58:33 +01:00
Ross Burton 45c0763fc9 Revert "wic/utils/partitionedfs.py: assemble .wic images as sparse files"
It turns out that dd's conv=sparse doesn't look at the file extents, but simply
checks if a "block" is all zero.  If the block of zero was meaningful it gets
lost and if the image is subsequently written to media using a sparse-aware
writer then the block of zeros won't be written at all.

This reverts commit 5fd592fbae2e046bcb8c3a6c3ef4993fe0400676.

(From OE-Core rev: 63d15764cc2014dba9fee2186f0c8b97c2ac5682)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-13 10:12:52 +01:00
Joshua Lock 29bc2f7423 wic/utils/partitionedfs.py: assemble .wic images as sparse files
The individual partitions created by wic are sparse but without
this change the assembled image is written as one (potentially
very) large file.

Preserve sparseness in the assembled image by passing the sparse
conversion symbol.

[YOCTO #9099]

(From OE-Core rev: 5fd592fbae2e046bcb8c3a6c3ef4993fe0400676)

Signed-off-by: Joshua Lock <joshua.lock@collabora.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-09 23:00:45 +01:00
Ross Burton f737af453d build-perf-test: add eSDK installed size to metrics
(From OE-Core rev: 57040305dea7e2167adb47c136a7b85f09ee7d24)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-09 23:00:45 +01:00
Randy Witt 0720425744 devtool: Create unlocked-sigs.inc containing items in the workspace
When a recipe is added to the workspace, the signatures for the tasks
will change. This means that bitbake must be told to allow the
signatures to be different if they are in locked-sigs.inc.

This is done by creating an unlocked-sigs.inc file which contains all
the recipes in the workspace each time devtool reads the workspace.

So not only will necessary things get added, previously added items will
be removed by virtue of them no longer being in the workspace.

This also makes sure that the extensible sdk picks up unlocked-sigs.inc
as part of the configuration.

[YOCTO #9195]

(From OE-Core rev: 6b2b5ffdcb8bf885a1c756ea132e9d2c55e13dcd)

Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-09 23:00:43 +01:00
Saul Wold 60786b8d90 runqemu-gen-tapdevs: Add note about NetworkManager & tap devices
NetworkManager can clobber tap devices if left alone, this gives a
note about how to set tap* as unmanaged

[YOCTO #8587]

(From OE-Core rev: 503a80de1cc9097fdc8f6b5b7d0e3a3991b0ecd2)

Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-06 22:57:26 +01:00
Robert Yang 298d875fac create-pull-request: fix for newer git
Fixed when git > 2.1.0:
$ ./scripts/create-pull-request -r HEAD^ -u contrib -b rbt/git
fatal: Not a valid revision: rbt/git
ERROR: git request-pull reported an error

This is because newer git requires both local and remote branch named as
rbt/git, but usually, we only named the remote branch as rbt/foo, and
foo for local branch.

Add a option '-l' to fix the problem, default is HEAD.

(From OE-Core rev: 98faa3ec872e06774b5870fcfb52f3ff91494779)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-06 22:57:26 +01:00
Humberto Ibarra d8f1f428bd scripts/oe-selftest: avoid the creation of coverage file when coverage not installed
Coverage subprocessing file is being created even when coverage
is not installed, which causes errors of "module not found" to
be send to the oe-selftest output.

This patch adds indent to the block of code creating this coverage
file, so it can only be executed when coverage is actually
installed.

[Yocto #9334]

(From OE-Core rev: 03d1711350895e941a476bd2d2a6cd389be07509)

Signed-off-by: Humberto Ibarra <humberto.ibarra.lopez@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-06 10:29:24 +01:00
Humberto Ibarra 6e5e225217 scripts/oe-selftest: remove coverage file if any coverage option is given
Coverage temporal file for sub-processing is being removed only
when the --coverage option was found. This is wrong since the
file is created when any coverage option (source, include or omit)
is given, even if --coverage is not one of them.

This patch makes sure to remove the file if any coverage option was
given.

(From OE-Core rev: 7157fd3bfbf2d5ae07fd4e726884baf7138bb24f)

Signed-off-by: Humberto Ibarra <humberto.ibarra.lopez@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-06 10:29:24 +01:00
Humberto Ibarra 5edfec4305 scripts/oe-selftest: remove unneeded coverage warning
There is a message that warns the user about enabling subprocessing
for coverage to work. After the fix for Yocto #8930,
this task is done automatically, so the warning is not needed
anymore.

(From OE-Core rev: 1ccfdb31ea8a50e695d368c4fc78185254aac763)

Signed-off-by: Humberto Ibarra <humberto.ibarra.lopez@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-06 10:29:23 +01:00
Otavio Salvador 672545b74b scripts/oe-buildenv-internal: Fix regression in BB_ENV_EXTRAWHITE setting
The commit OE-Core:ada4639 (oe-buildenv-internal: simplify derivation
of BB_ENV_EXTRAWHITE) changed the format of BB_ENV_EXTRAWHITE
variable to:

,----
| BB_ENV_EXTRAWHITE='ALL_PROXY
| BB_NO_NETWORK
| BB_NUMBER_THREADS
| ...
| '
`----

Instead of:

,----
| BB_ENV_EXTRAWHITE='ALL_PROXY BB_NO_NETWORK BB_NUMBER_THREADS...'
`----

The old format allow for external script parsing easier and there is
no need to change the format as it has no benefit from usage
perspective.

(From OE-Core rev: 773e9ef9bacbf1e63fe758915d599d11762b8c1f)

Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-06 10:29:23 +01:00
Humberto Ibarra f4f1d206f4 scripts/lib/bsp/help.py: Typo in help for yocto-bsp create
Fix typo of the word "parameter" in for 'yocto-bsp create'.
Typo appears in both, usage and help.

Also, the word "description" is mispelled.

[Yocto #9282]

(From meta-yocto rev: 12c7243abd91b374b1b62c6a1ad13b0d25aa0e4c)

Signed-off-by: Humberto Ibarra <humberto.ibarra.lopez@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-05 14:33:58 +01:00
Paul Eggleton 4d364f27e7 classes/buildhistory: optimise getting package size list
Invoking oe-pkgdata-util in turn for every package in the list was slow
with a large image. Modify oe-pkgdata-util's read-value command to take
an option to read the list of packages from a file, as well as prefix
the value with the package name; we can then use this to get all of the
package sizes at once. This reduces the time to gather this information
from minutes to just a second or two.

Fixes [YOCTO #7339].

(From OE-Core rev: 51c24904cc1bc823bccc3f179b8d7a192dace168)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-05 14:33:57 +01:00
Ioan-Adrian Ratiu 31a10cb6af wic/isoimage-isohybrid.py: change cpio generated uid&gid to root
By default cpio preserves the uid&guid's of the original user which
leads to host contamination and boot failures because commands like
mount from initramfs expect to be run by root and the original host
user might not even exist on the target.

(From OE-Core rev: 28910ee2eacc15cf42b5e58bd43b3bd15c34eb97)

Signed-off-by: Ioan-Adrian Ratiu <adrian.ratiu@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-03 15:51:36 +01:00
Ioan-Adrian Ratiu 5cabf3b061 wic/isoimage-isohybrid.py: use glob to find initramfs location
Some filenames can omit 'initramfs', or use other names. This makes
detection more flexible by using only the image name, machine arch and
image type in a glob wildcard.

(From OE-Core rev: ca516f5907a661606c35e1ca5c2ece9fc79c77ea)

Signed-off-by: Ioan-Adrian Ratiu <adrian.ratiu@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-03 15:51:36 +01:00
Humberto Ibarra 965c72c38b yocto-bsp: Set correct default branches and branches base for i386, qemu and x86_64 archs
Kernel recipes for linux-yocto_4.* and greater have outdated branches as default,
making it impossible to find the right branch if the user picks the default value.
The branches_base property uses these outdated branches also.

This updates standard/common-pc and standard/common-pc-64 branches to standard/base.

The fix was tested using 'yocto-bsp create' with each one of the following
archs:

-i386 with kernels 4.1 and 4.4
-x86_64 with kernels 4.1 and 4.4
-qemu (i386 and x86_64) with kernels 4.1 and 4.4

After the layer was created, it was added to local.conf and the MACHINE was set
accordingly.

'bitbake linux-yocto' ran successfully with each configuration tested.

[YOCTO #9160]

(From meta-yocto rev: d471e3dd7c5080a29f64b60b554f17ee706ee772)

Signed-off-by: Humberto Ibarra <humberto.ibarra.lopez@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-01 07:14:43 +01:00
Richard Purdie 51217057d1 scripts, lib: Don't limit traceback lengths to arbitrary values
There appears to have been a lot of copy and pasting of the code
which prints tracebacks upon failure and limits the stack trace to
5 entries. This obscures the real error and is very confusing to the user
it look me an age to work out why some tracebacks weren't useful.

This patch removes the limit, making tracebacks much more useful for
debugging.

[YOCTO #9230]

(From meta-yocto rev: 2f00b2605353b9757a40636870ad20ee70bcab9d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-31 23:01:38 +01:00
Juro Bystricky 4dde12f17b relocate_sdk: additional error checks
When installing SDK in a non-default location and the path length
of the SDK install location is longer than the path length of the
default SDK location, relocation of .ldsochache section will overwrite
file location outside of the .ldsocache section size.
In addition, additional checks were added to ensure that any
path in sections .gccrelocprefix and .ldsochache will not exceed
the space allocated for it within the file, which would also result
in file corruption.

[YOCTO #9268]

(From OE-Core rev: 4d949da965a99ab33798af49e5584c8bb9f0f626)

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-31 23:01:37 +01:00
Paul Eggleton 4b7790915f devtool: modify: call shutdown on tinfoil when done
Strictly speaking we ought to explicitly shut down a tinfoil instance
when we're done with it. This doesn't affect modify's operation but is
important if you want to be able to call into modify() from another
plugin (though anyone doing so should be advised that the function is
by no means a stable API and is subject to change in future releases).

(From OE-Core rev: 626dbadf22b57a22a8f8b9d1957937120f4ba4d5)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-31 23:01:37 +01:00
Paul Eggleton 2aec71e028 oe-publish-sdk: exclude sstate-cache if publishing minimal SDK
If SDK_EXT_TYPE is set to "minimal" then the SDK won't contain many
sstate artifacts, and you're required to set up an sstate mirror in this
case anyway so there's no point publishing the "stub" sstate-cache
directory from within the SDK since it won't be useful for update
purposes and may be confused with the real sstate-cache.

There is however a possibility that people might publish the real
sstate-cache directory under the same output directory provided to
oe-publish-sdk, thus deleting it after extracting (as we were doing with
other files we wanted to clean up at the end) would be problematic,
besides which extracting it and then deleting it is wasteful. Thus,
introduce a "-p" command line option to the SDK installer that we can
use to tell tar not to extract the items we don't want when publishing.
This has the added benefit of mostly keeping references to these in the
place they belong i.e. in populate_sdk_ext.bbclass.

(From OE-Core rev: 774b85d42db1d81936d4e4af4f6fb2c57cb51d2c)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-31 23:01:37 +01:00
Paul Eggleton 8ef70163fc oe-publish-sdk: prevent specifying a directory for the SDK argument
The SDK argument is expected to be an installer .sh file; if a directory
is specified we can get an ugly failure later on; best to check up
front.

Fixes [YOCTO #9065].

(From OE-Core rev: ce71f5c2fb8a7b473988da30bbb9bec95e8a6f5e)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-31 23:01:37 +01:00
Richard Purdie c37d5426b1 scripts, lib: Don't limit traceback lengths to arbitrary values
There appears to have been a lot of copy and pasting of the code
which prints tracebacks upon failure and limits the stack trace to
5 entries. This obscures the real error and is very confusing to the user
it look me an age to work out why some tracebacks weren't useful.

This patch removes the limit, making tracebacks much more useful for
debugging.

[YOCTO #9230]

(From OE-Core rev: 6069175e9bb97ace100bb5e99b6104d33163a3a2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-31 23:01:36 +01:00
Robert Yang c093f7c623 runqemu: fix for iso
It should be the similar type as hddimg, rather than ext234 or btrfs.

(From OE-Core rev: d5ddc5ec1628c94bd5edc45bc821da1ce616e80f)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-29 23:20:12 +01:00
Humberto Ibarra 3a934a8087 scripts:/oe-selftest: Use timestamp instead of test names in coverage data file
This fixes the problem by changing the name to the coverage data file,
using the timestamp as an identifier.

The name for the coverage data file is constructed based on the tests
ran; this has created a couple of issues so far, affecting coverage report.

If --run-tests-by option is given, the data file name won't have any
identifier, causing following runs to overwrite themselves. On the
other hand, if too many tests are given, the file name exceeds linux
limits and fails to store the coverage data all together.

[Yocto #9253]

(From OE-Core rev: 7c0c9d1411e04935675f4c4be8768723c1d09b50)

Signed-off-by: Humberto Ibarra <humberto.ibarra.lopez@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-28 15:55:50 +01:00
Wenlin Kang c3c793b428 relocate_sdk: fixed .gccrelocprefix section handling
When fixing paths for .gccrelocprefix section, it will corrupt the next
entry during updating the current one if "new_prefix" length is more
than "DEFAULT_INSTALL_DIR", this problem is obvious on the code, but it's
only found when install sdk onto a net file system.

(From OE-Core rev: c43192c46bb1759bf6909190b89d3b60918777c5)

Signed-off-by: Wenlin Kang <wenlin.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-28 15:55:50 +01:00
Jussi Kukkonen cef8bc9e80 gio-module-cache: Add class for Gio modules
The new class uses gio-querymodules tool on postinst and postrm.
This regenerates the module cache which is useful to avoid loading
modules that are not needed at runtime: If a Gio module is not
listed in the cache file it will always get loaded.

* Add a postinst-intercept 'gio-module-cache': it runs
  gio-querymodules using qemuwrapper. This is required because the tool
  actually loads the modules to generate the cache.
* Add a gio-module-cache class that adds postinstall and postrm
  scripts. In the sysroot population case use the new intercept.
* Inherit the new class in glib-2.0, glib-networking and gconf.

Fixes [YOCTO #9241].

(From OE-Core rev: 168cf5502a2dda78348cc9679e37bed0c69c0208)

Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-28 15:55:49 +01:00
André Draszik 9ac1b6fc39 oe-git-proxy: support username / password in http proxy
We also make sure to correctly support usernames that contain spaces.

For simplicity sed + regex has been replaced with shell parameter expansion,
which works in both, bash and dash.

(From OE-Core rev: ce4ff3dd80379e1bf71a967e0512591aaa046308)

Signed-off-by: André Draszik <adraszik@tycoint.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-28 15:55:49 +01:00
André Draszik a15541dc27 oe-git-proxy: also check all_proxy and http_proxy env variables
(From OE-Core rev: 00fe3004aa624afee62315d5ea7566853b926931)

Signed-off-by: André Draszik <adraszik@tycoint.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-28 15:55:49 +01:00
Ed Bartosh 5cb7705d05 wic: fix type of no-table option
Type of --no-table option was incorrectly set in new wks parser.
It causes parser to require argument for this option, which makes
wic to fail with wks files that use --no-table:
    Error: argument --no-table: expected one argument

Changed action parameter to 'store_true' to fix the issue.

(From OE-Core rev: d483724cf3515f76e1b798a2018e2f3fa2bad0ba)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-28 15:55:49 +01:00
Andre McCurdy 33a6135294 oe-buildenv-internal: simplify derivation of BB_ENV_EXTRAWHITE
Use 'sort --unique' to combine the two lists and remove duplicates.

(From OE-Core rev: ada4639f1e2952e144ea7dfb2a784181d7fcc96a)

Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-28 15:55:48 +01:00
Humberto Ibarra 1dd9348008 scripts/lib/bsp/help.py: Add missing options to yocto-bsp help and usage
Add the options --codedump and --skip-git-check to the yocto-bsp help and
yocto-bsp usage, since they are currently missing.

[YOCTO #8322]

(From meta-yocto rev: dfdf97b4239639affc8ce22e338a291d4b0dfc76)

Signed-off-by: Humberto Ibarra <humberto.ibarra.lopez@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-26 08:06:36 +00:00
Robert Yang b47c53b70e runqemu-internal: split the code into functions
Use config_<machine> to split the code into separate functions, so that
different machines won't affect each other, and they will have a better
structure.

(From OE-Core rev: 6098a72e8d21f6321636e727703ca7c29d9af142)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-25 10:29:14 +00:00
Robert Yang fae732f24e runqemu-internal: cleanup unsed code
* remove akita and spitz related code
  They are not supported by runqemu anymore:
  $ runqemu spitz
  Error: unable to classify arg [spitz]
  So remove related code.

* Remove checking of 256M for qemuarm, qemu can check it, for example:
  $ runqemu qemuarm qemuparams="-m 1024"
  [snip]
  qemu: Too much memory for this machine: 1024 MB, maximum 256 MB
  [snip]

(From OE-Core rev: 36fb785bf8cd2f387c91d52f597602a5dbc3948b)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-25 10:29:14 +00:00
Robert Yang e469bb722e runqemu: simplify checking for iso and ramfs
(From OE-Core rev: 69a1fca4374797dea56035ce56a17441a2ca9280)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-25 10:29:14 +00:00
Robert Yang 3610329929 runqemu: add support for qcow2 and vdi
[YOCTO #9168]

(From OE-Core rev: 0f8306c77b4ebed1ff127b0786b7109abf0d57cd)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-25 10:29:14 +00:00
Robert Yang d85ca4a616 runqemu: remove ISO and RAMFS from help text
They don't work, and the script can check the type correctly.

(From OE-Core rev: b5cc1e70dbd5df160ddedcaa40d0ab714a307561)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-25 10:29:14 +00:00
Robert Yang 58bc8542de runqemu: simplify the checking for vm images
* So that we can add more image support easliy.
* I think that wic should be vm images.

(From OE-Core rev: 82d0014a0e1526ffa1ff7c8ea3903aeae31bada4)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-25 10:29:14 +00:00
Robert Yang 6716eb245d runqemu: fix ROOTFS for vmdk
* Make it can boot scsi and virtio block drive such as root=/dev/sdX and
  /dev/vdX.

* Drop VM from help info, id doesn't work, and the script can check
  whether it is a vm disk or not.

* Make it can be run by:
  $ runqemu tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.vmdk
  or:
  $ runqemu qemux86-64 vmdk

[YOCTO #9170]

(From OE-Core rev: 88c081b10902ec52928be78ad320c474bb918e01)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-25 10:29:14 +00:00
Paul Eggleton 13a31b15ec oe-publish-sdk: fix remote publishing
A typo in the command prevented the last part of the publish from
succeeding.

(From OE-Core rev: 5d51bd1f02fb20e02d0de0ac9874d7f9a6f5ddc6)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-25 10:29:14 +00:00
Paul Eggleton 9926425bbd oe-publish-sdk: improve help output slightly
This was just copy-and-pasted from devtool - set a reasonable
description and tweak the sdk parameter help text.

Also add a copyright statement and drop the opening comment describing
the command-line syntax that duplicated the help output (with at least
one mistake in it).

(From OE-Core rev: b18f9cb62e12d068fb840fd7ab9f35b4f039e649)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-25 10:29:13 +00:00
Paul Eggleton 905286ce46 oe-publish-sdk: drop SDK installer file from published output
We don't really need the SDK installer in the published output, for two
reasons:

1) The directory produced is for the consumption of devtool sdk-update,
   and the installer isn't used by that at all
2) It wouldn't really make sense to point users at the update directory
   to download the SDK installer because it contains a bunch of things
   that aren't meant for manual download, so it wouldn't be very tidy.

Leaving the file present can mislead you into thinking the opposite of
both of the above.

(From OE-Core rev: a9ec72b7897ffc0b901c1ccbcbe3cabbc7ac41ee)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-25 10:29:13 +00:00
Paul Eggleton 0523378082 devtool: add: create git repository if URL specified as positional argument
When we add from a fetched URL we are supposed to turn the resulting
source tree into a git repository (if it isn't already one). However, we
were using the older deprecated option name here instead of the
positional argument, so "devtool add -f <url>" resulted in the repo
being created but "devtool add <url>" didn't, which was wrong.

Also update the oe-selftest tests to check that this worked.

(From OE-Core rev: a7b6b1f8cc1c096724f794ac9dee312b0f771f66)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-25 10:29:13 +00:00
Paul Eggleton 11c1d30152 devtool: add: delete externalsrc files on npm recipe do_install
The npm class just installs whatever is in ${S}; if you're using
externalsrc in conjunction with it the symlinks (oe-workdir and oe-logs
by default) plus singletask.lock will end up in the final package, which
isn't really right. Introduce a variable so we know the path the files
will be installed into within npm.bbclass, and append to do_install
within the workspace bbappend to delete the files.

(From OE-Core rev: 766845e06db9d7d595e836ea1364c16fa132a413)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-25 10:29:13 +00:00
Paul Eggleton 552a68ad1c devtool: configure-help: fix error if do_configure not already run
The code here for running do_configure if it hadn't already been run was
using the wrong string substitution parameters; fix it and test it.

(From OE-Core rev: b2677a4448dbc42e523c731b953b44006749252c)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-25 10:29:13 +00:00
Paul Eggleton 06845723d4 devtool: sdk-install: add option to allow building from source
By default the sdk-install subcommand expects to restore the requested
items from sstate and fails if it can't. If the user is OK with building
from source, add a -s/--allow-build option to allow them to do that. In
the process, ensure we show the status output while we're installing.

Also add the missing header to the top of the file.

(From OE-Core rev: a86b426cdd465ec5cb08bb5fa7729e4e673d94bb)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-25 10:29:13 +00:00
Peter Kjellerstedt dab6d59e47 oe-buildenv-internal: Correct the sed expression which updates $PATH
Without this, the code that adds paths to $PATH could cause it to end
up with a trailing : which would then cause an error from the sanity
checker.

(From OE-Core rev: 9c200760cbbe322ed884729eb395f389c863e1c8)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-21 11:57:31 +00:00
Peter Kjellerstedt 76f10fd046 oe-buildenv-internal: Some clean up
* Consistent indentation (four spaces)
* Use [ -z ...] and [ -n ... ] where possible
* Unset temporary variables
* Use $(...) instead of `...`
* Avoid an unnecessary call to expr

(From OE-Core rev: 791eec016792c3f4c04b12ae6ff93c1e23266f87)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-20 23:12:30 +00:00
Peter Kjellerstedt 4d1efc38ca oe-buildenv-internal: Add variables individually to BB_ENV_EXTRAWHITE
Instead of adding all variables to BB_ENV_EXTRAWHITE as one, treat
them separately and add them one by one as needed.

(From OE-Core rev: 516b63fd9dea6fcc304fca920206467d2565dede)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-20 23:12:30 +00:00
Peter Kjellerstedt 39ac332f66 oe-buildenv-internal: Add paths to $PATH individually
Instead of assuming that the path to the scripts directory always is
in $PATH directly before the bitbake directory, treat them as separate
paths and add them individually to $PATH.

(From OE-Core rev: 5b9e91a3b1d4f56c8646d7d7937ab453cc9c40a0)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-20 23:12:30 +00:00
Daniel Istrate ee4f61b7e1 oe-selftest: Fixed --list-tests-by tag option
Commit 35be679513 broke the
--list-tests-by tag option.
This patch fixes that.

Having a module in lib/oeqa/selftest named testmodule:
class TestClass(oeSelfTest):
    @tag(feature='tag1')
    def test_func1(self):
        pass
    @tag(feature=('tag1', 'tag2'))
    def test_func2(self):
        pass
    @tag(feature=('tag2', 'tag3'))
    def test_func3(self):
        pass
    @tag(feature=('tag1', 'tag2', 'tag3'))
    def test_func4(self):
        pass

$ oe-selftest --list-tests-by tag tag1
 ID  TAG(s)            NAME        CLASS      MODULE
----  ----------------  ----------  ---------  --------
      tag1              test_func1  TestClass  testmodule
      tag1, tag2        test_func2  TestClass  testmodule
      tag1, tag2, tag3  test_func4  TestClass  testmodule
______________________________
Filtering by:	 tag
Looking for:	 tag1
Total found:	 3

$  oe-selftest --list-tests-by tag tag1 tag2
  ID  TAG(s)            NAME        CLASS      MODULE
----  ----------------  ----------  ---------  --------
      tag1              test_func1  TestClass  testmodule
      tag1, tag2        test_func2  TestClass  testmodule
      tag1, tag2, tag3  test_func4  TestClass  testmodule
      tag2, tag3        test_func3  TestClass  testmodule
______________________________
Filtering by:	 tag
Looking for:	 tag1, tag2
Total found:	 4

$ oe-selftest --list-tests-by tag tag*
  ID  TAG(s)            NAME        CLASS      MODULE
----  ----------------  ----------  ---------  --------
      tag1              test_func1  TestClass  testmodule
      tag1, tag2        test_func2  TestClass  testmodule
      tag1, tag2, tag3  test_func4  TestClass  testmodule
      tag2, tag3        test_func3  TestClass  testmodule
______________________________
Filtering by:	 tag
Looking for:	 tag*
Total found:	 4

(From OE-Core rev: 28c1cffacf341ad64a5b68d8a0176f92b49135c0)

Signed-off-by: Daniel Istrate <daniel.alexandrux.istrate@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-20 23:12:29 +00:00
Markus Lehtonen 51464e7b89 devtool: change config symlink name to .config.new
Otherwise (if the symlink is named .config) kernel build considers
source tree as dirty and fails.

[YOCTO #9270]

(From OE-Core rev: d36699b28c661880957d744420df9e23cdd76957)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-20 23:12:27 +00:00
Paul Eggleton 600b7007f6 recipetool: create: don't create extra files directory unconditionally
The extra directory next to the recipe should only be created if there
are files to put into it; currently only the npm plugin does this. I
didn't notice the issue earlier because the test was actually able to
succeed under these circumstances if the recipe file came first in the
directory listing, which was a fault in my original oe-selftest test;
apparently on some YP autobuilder machines the order came out reversed.

With this change we can put the oe-selftest test that highlighted the
issue back to the way it was, with an extra check to reinforce that only
a single file should be created.

(From OE-Core rev: b8b778345eb0997c2cd952a1f61fdd2050b6b894)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-20 23:12:25 +00:00
Humberto Ibarra 9dbfbe9940 scripts/oe-selftest: Add short names to most common options
Add short names to most common options in oe-selftest. The options
changed were --run-tests, --run-all-tests, --list-tests and
--list-modules.

[Yocto #9079]

(From OE-Core rev: 05f699e3c2847f0c64118df8ffd43bf2f95ed4d1)

Signed-off-by: Humberto Ibarra <humberto.ibarra.lopez@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-12 22:11:48 +00:00
Ross Burton 9e69963f11 python: move ast module into python-core
The compile() method returns objects that inherit from ast.AST so it's best that
python-core contains this class.

[YOCTO #8684]

(From OE-Core rev: 48bc643a4ebe74d0fa49fedbe7b0fd63fd0003e8)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-12 22:11:47 +00:00
Humberto Ibarra 35be679513 scripts/oe-selftest: Add search expression matching to run/list options
The oe-selftest script required an exact matching for the parameters
passed to its run-tests-by and list-tests-by options. Many tests
can be retrieved here and filtering is a must.

This patch add this filtering functionality by enabling the use
of wildcards such as "*".

[Yocto #8916]

(From OE-Core rev: 01a242645cc12daed3f4da30d8c49813642d75e6)

Signed-off-by: Humberto Ibarra <humberto.ibarra.lopez@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-12 22:11:47 +00:00
Paul Eggleton bc0e99d2b1 recipetool: create: shrinkwrap and lockdown npm modules
"npm shrinkwrap" creates a file that ensures that the exact same
versions get fetched the next time the recipe is built. lockdown is
similar but also includes sha1sums of the modules thus validating they
haven't changed between builds. These ensure that the build is
reproducible.

Fixes [YOCTO #9225].

(From OE-Core rev: 277377f13b2b771915eb853e336ca24b84523ed1)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 17:00:29 +00:00
Paul Eggleton 309b2e6c57 recipetool: create: support creation of additional files by plugins
Allow plugins to create additional files to go alongside the recipe. The
plugins don't know what the output filename is going to be, so they need
to put the files in a temporary location and add them to an "extrafiles"
dict within extravalues where the destination filename is the key and
the temporary path is the value.

devtool add was also extended to ensure these files get moved in and
preserved upon reset if they've been edited by the user.

(From OE-Core rev: 334b9451111b7e3efbb43b3a4eecebcab8ec6f0e)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 17:00:29 +00:00
Paul Eggleton 2279eb2a4c recipetool: create: check if npm available if npm:// URL specified
If the user specifies an npm:// URL then the fetcher needs npm to be
available to run, so check if it's available early rather than failing
later.

(From OE-Core rev: a08d12ad867c292f7474731a0fe5e51e712446d6)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 17:00:29 +00:00
Paul Eggleton 91455005b6 recipetool: create: split npm module dependencies into packages
Rather than rolling all of an npm module's dependencies into the same
package, split them into one module per package, setting the SUMMARY and
PKGV values from the package.json file for each package. Additionally,
mark each package with the appropriate license using the license
scanning we already do, falling back to the license stated in the
package.json file for the module if unknown. All of this is mostly in
aid of ensuring all modules and their licenses now show up in the
manifests for the image.

Additionally we set the main LICENSE value more concretely once we've
calculated the per-package licenses, since we have more information at
that point.

(From OE-Core rev: 8226805f83d21e7c1d2ba21969f3e8ee4b137496)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 17:00:29 +00:00
Paul Eggleton d46827cfd3 recipetool: create: add license file crunching
Matching license texts directly to md5sums only goes so far. Some
licenses make the copyright statement an intrinsic part of the license
statement (e.g. MIT) which of course varies between projects. Also,
people often seem to take standard license texts such as GPLv2 and
reformat them cosmetically - re-wrapping lines at a different width or
changing quoting styles are seemingly popular examples. In order to
match license files to their actual licenses more effectively, "crunch"
out these elements before comparing to an md5sum. (The existing plain
md5sum matching has been left in since it's a shortcut, and our list of
crunched md5sums isn't a complete replacement for it.)

As always, this code isn't providing any guarantees (legal or otherwise)
that it will always get the license correct - as indicated by the
accompanying comments the LICENSE values it writes out to the recipe are
indicative and you should verify them yourself by looking at the
documentation supplied from upstream for the software being built if you
have any concerns.

(From OE-Core rev: 553bb4ea5d51be5179e7d8c019740cf61ece76ea)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 17:00:29 +00:00
Paul Eggleton 3fd244b94f recipetool: create: match *LICENSE* as a license file
For example, this picks up a file named MIT-LICENSE.txt.

(From OE-Core rev: 103b4d26b340cbdf70bf43906e293f3497671fdc)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 17:00:29 +00:00
Paul Eggleton 2b6a35212d recipetool: create: improve mapping for autotools program macros
Make the following improvements to mapping items specified in
AC_CHECK_PROG, AC_PATH_PROG and AX_WITH_PROG to recipes/classes:

* Produce a map of native recipe -> binary for all binaries currently in
  STAGING_BINDIR_NATIVE and use this when mapping items
* Add some more entries to the class map
* Ignore autotools binaries since they are covered by the inherit of
  autotools
* Ignore coreutils-native since that would almost always be a bogus
  dependency

(From OE-Core rev: 5614c5ae6a004d4367eccc34dd3cc7ee61fb7e57)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 17:00:29 +00:00
Paul Eggleton 1607fac521 recipetool: create: be more tolerant of spacing in configure.ac
Allow for whitespace in appropriate places, and ensure we match all
whitespace chars not just the space character.

(This fixes extracting dependencies from tmux's configure.ac, for
example.)

(From OE-Core rev: 63524ac8093b734aa4f29f4ea47bcc036f748314)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 17:00:29 +00:00
Mariano Lopez a27ca6da59 yocto-bsp: Update templates to 4.4 kernel
Add kernel 4.4 support. kernel 4.1 was left in the
tree for user to choose between 4.1 and 4.4.

[YOCTO #9047]

(From meta-yocto rev: c04093ec9b3cd151cf857bca9ad35bc9a0d4a4b1)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 16:58:14 +00:00