Generators in Python 3 don't have .next method. It's recommended
to use 'next' builtin instead. As it also present in Python >= 2.6
it should make wic code compatible with both Python 2 and Python 3.
[YOCTO #9412]
(From OE-Core rev: 9b7ab632e47d786dd979262015dbfb1254103f83)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Removed check for unicode type as it doesn't work in Python 3.
This check is not needed for wic as all its output seem to be
strings. This allows to run code under both pythons.
[YOCTO #9412]
(From OE-Core rev: a56924b4a0102e401b5e37d857a08bab15da974e)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
stdeout and stderr content returned by communicate API has different
types in Python 3(bytes) and Python 2(string). Decoding it to 'utf-8'
makes it unicode on both pythons.
Decoded stdout and stderr output to utf-8 to make the code
working under both Python 2 and Python 3.
(From OE-Core rev: 5b556f58a171e3d45107bb56a1f780e5c1abba37)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Division operator works differently in Python 3. It results in
float unlike in Python 2, where it results in int.
Explicitly used "floor division" operator instead of 'division'
operator. This should make the code to result in integer under
both pythons.
[YOCTO #9412]
(From OE-Core rev: 997ff239bd753a7957cc14c6829b2f093d9bcef6)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As bitbake is not ported to Python 3 yet it's better to
avoid using its APIs as much as possible to be able to
test wic under Python 3 at least partially.
Used distutils.spawn.find_executable API in favor of
bb.utils.which to get path of the command to run.
(From OE-Core rev: 9658956bf8a5da779e06f71941de9b3e89415cdc)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This suffix is not supported by Python 3. Wic code works
without it on Python 2 too, so it's safe to remove it.
[YOCTO #9412]
(From OE-Core rev: 296db7e33bd71585cac63dc78c2c95bc619b4a86)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Wic plugin machinery implemented using metaclasses.
Reimplemented plugin machinery using this advice from
https://wiki.python.org/moin/PortingToPy3k/BilingualQuickRef
Syntax for creating instances with different metaclasses is very
different between Python 2 and 3. Use the ability to call type instances
as a way to portably create such instances.
Now it should work under both Python 2 and Python 3.
[YOCTO #9412]
(From OE-Core rev: e62fe5a41bdcdd72b9b257fecff7ccdc59c76d33)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Removed 'raw', 'ask', 'choice' and 'pause' functions from
msger.py as they're not used in wic code and some of them
use raw_input, which is not present in Python 3.
[YOCTO #9412]
(From OE-Core rev: eb87d591ef67f1953b2689430ef6c5a6a27a5b6e)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
New syntax 'except Exception as err' is supported by Python >= 2.7.
Old syntax 'except Exception, err' is not supported by Python 3.
Used new syntax to be able to run wic on Python 3.
[YOCTO #9412]
(From OE-Core rev: 15e88714d6b0a93f72e8a19b083fcc1f2006e128)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
'with' statement is not used in baseimager.py
It's supported by Python 2.7, which is included into all target
distros. Other wic modules use this statement.
Removed useless 'from __future__ import with_statement' from
wic code.
(From OE-Core rev: 528a1f20939589949831efbb4de6336776efe7d5)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
All classes in Python3 are new style classes and don't need
to be inherited from object. Wic code works fine without
this inheritance even with Python2, so it's harmless to
remove it.
[YOCTO #9412]
(From OE-Core rev: a146b03ee7d0aa5bc1722da5977a5952782b69bf)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Replaced calls of dict.keys and dict.has_key methods with the
'key in dict' statement. 'key in dict' is more pythonic, faster
and readable. dict.has_key doesn't exist in Python 3.
[YOCTO #9412]
(From OE-Core rev: 003df7dfb932c551953fbf1bd769b3c31bd16fb4)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Dictionary method 'iteritems' doesn't exist in Python 3.
Replaced 'iteritems' with 'items' to be able to run the
code under both Python 3 and Python 2.
[YOCTO #9412]
(From OE-Core rev: 5b14eb8d68aaca82de4f8f6bcb28ad6f4a5125d0)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Print statements have been replaced with print function in
Python 3. Replaced them in wic code to be able to run it
under both Python 2 and Python 3.
[YOCTO #9412]
(From OE-Core rev: ee6979a19c77931c3cf6368e695e370d46192fef)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As suspected, invoking "git archive" with all intended files as
parameters can run into command line length limitations. Splitting up
the parameters into multiple invocations (xargs-style) works and was
tested after encountering the situation in practice.
(From OE-Core rev: 1cb484ab99eabb5c24792757ab09d7f170f2e614)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the "history" option is not set in the combo-layer.conf, the
intended default was to use the traditional method. Passing "True" as
default when querying the config was unintentional.
Also remove some left-over debugging code.
(From OE-Core rev: d0304acb05b926b08805d8652e12eaf19bf53ad6)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If an image with the filename foo.bb could be built using the name "bar"
instead, then build-sdk would fail to create the derivative sdk.
This was because the code assumed that the file name matched the target,
which is not necessarily the case.
(From OE-Core rev: d58a326b6960be14b8a049253559aec9582b7d0d)
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The core idea is that all commits get imported, including merge
commits, and joined into one big merge commit that imports the changes
from the individual components into the main branch of the combined
repository.
This is done by copying the files in each commit and removing deleted
ones, instead of trying to patch the combined repository.
The advantages of doing updates in this mode are:
- works for arbitrary upstream repos, not just those which
support conversion into a linear set of patches
- listing history shows that commits where developed
independently in the different components, instead of
artificially showing them as if they had been developed
one after the after (component "aaaa" before "bbbb", then "ccc", ...)
- bisecting becomes easier: when upstream repos only ensure consistency
when merging into their "master" branches, then those merge
commits are good candidates for test builds also in the combined
repo
- more data mining can be done, for example showing who merged a commit
and when
Selecting a subset of the files is supported, albeit with a slight
different semantic for wild card matching compared to other code paths
(/ is matched by * and ?). Empty commits get skipped because typically
they are a result of filtering (but that is not checked, so
intentionally empty commits also get skipped).
Other code paths are intentionally left unchanged, to avoid
regressions. However, the downside is that some opportunities for
refactoring (in particular regarding file filtering) were ignored.
(From OE-Core rev: 660f76b6fb0fb95738a2c8f50e0a99ffa5831c64)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow setting the environment. Due to a subprocess quirk, it must
always be set explicitly (reuses the one from the previous call if not
set, instead of falling back to os.environ).
Embedding nul characters will be useful for parsing git output more
reliably; support dumping such output a bit better.
(From OE-Core rev: 0af4fadafce690fc8357196cf7247bd222c08d10)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When setting "history = True" in combo-layer.conf consistently for the
components involved in an update or using "update" together with the
"--history" command line flag, a new mode for updating will be used
that does not rely on exporting/importing patches.
A config setting is used because it should be used consistently by
everyone using the same config, without having to remember to use an
additional command line parameter.
There are no real global settings, so the setting is checked
separately for each component although the setting has to be set
consistently. This restriction could be removed later.
In practice, putting "history" into the "[DEFAULT]" section is the
easiest approach for configuring it.
The actual code changes split up action_update and the
combo-layer.conf handling in preparation for this new mode, without
implementing the mode itself.
(From OE-Core rev: c9dab31f5f6dc225f5c2c2ca3ec9aeab2ff655d5)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When a branch is re-used, the kernel tools turns off any patch pushing unless
'mark patching' is explicitly set.
[YOCTO #9120]
(From meta-yocto rev: 427f5473722e15e288cbce251a9ce18989c23548)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This gives us a progress bar for the image write, which is quite helpful. See
https://www.ivarch.com/programs/pv.shtml.
(From OE-Core rev: 5b20cb2db929daaf0bf81c05368174e9c364ffab)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix two problems falling back to the "license" field from package.json
when no license file is present:
1) The function that was supposed to return the license field value was
always explicitly returning None, and this was never noticed (because
the test cases never exercised the fallback as they provided license
files for each module).
2) Fix the main package not falling back because it had a default of an
empty list, which evaluates to '' instead of 'Unknown'.
(From OE-Core rev: 59381a9450949ce6b4b03adb717e950b999830f3)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This aligns with our existing 'optional arguments' to 'options' change, and
seems more intuitive for users.
(From OE-Core rev: 8a1cd471210e5fb77952f28172084bf6a4fb73e8)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There's no need to iterate over the action groups here, as self._optionals and
self._positionals are available.
(From OE-Core rev: 408694f4320f3cb52a391e5b927fb8c8ba16c1d2)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As an example, `recipetool create foo bar baz` shows `recipetool: error:
unrecognized arguments: bar baz` and then displays the main help, not the help
for the create command. Fix by saving the subparser name and using it in
parse_args() to look up the subparser.
(From OE-Core rev: 7fdaaedf4c63c8d019f03f84e22f9b838ef19aa6)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This aligns our subclassed error() with that in the original class, using
_print_message and self.prog. Also add a docstring based on the original.
(From OE-Core rev: cf0c5175136966eefde8c0d9aa0679e85779f713)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Used sparse_copy API in favor of dd/cp in rawcopy plugin to
preserve sparseness of the copied raw content.
[YOCTO #9099]
(From OE-Core rev: 04eca59068a79ae6a9969be495c4cdf0c5c3e466)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Copied partition images into final partitioned image using
sparse_copy API. This method preserves sparseness of the
final image. It also makes wic much faster, as unmapped
blocks of the partition images are not copied.
[YOCTO #9099]
(From OE-Core rev: 7f21427aca5df81d8881027fd98f71b821cf31d7)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to make wic images sparse sparse_copy function has been
copied from meta-ostro:
https://github.com/kad/meta-ostro/blob/master/meta-ostro/lib/image-dsk.py
This function uses filemap APIs to copy source sparse file into
destination file preserving sparseness.
The function has been modified to satisfy wic requirements:
parameter 'skip' has been added.
[YOCTO #9099]
(From OE-Core rev: bfde62bdc03152a4d3d383512479b974fa867f94)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to make wic images sparse set of APIs has been copied
from bmap-tools project.
filemap.py module is taken from bmap-tools project:
https://github.com/01org/bmap-tools/blob/master/bmaptools/Filemap.py
It implements two ways of get information about file block: FIEMAP
ioctl and the 'SEEK_HOLE / SEEK_DATA' features of the file seek
syscall.
Note that this module will be removed as soon as bmaptool utility
supports copying sparse source file into destination file (this is
already agreed with the maintainer of bmap-tools project).
[YOCTO #9099]
(From OE-Core rev: 182639ddc9cda85c896a54c1c64fd1fb145071a1)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Moved DiskImage class from utils/fs_related.py to
imager/direct.py as it's only used there.
Removed fs_related module as it doesn't contain anything
except of DiskImage.
(From OE-Core rev: b3cc471790784c28f9362fcd6fc6a81c4316754c)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Removed fs_related.makedirs as is not used anywhere. The name is
easy to confuse with os.makedirs.
(From OE-Core rev: 796b114863ef20fbc89da45dbe6780abe1256f5e)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is no need in this inheritance as DiskImage class
is used only in one module and no other classes are inherited.
(From OE-Core rev: 5af1d9bedc2c961eb91faf80251f24c3df754d76)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Used truncate instead of dd to create wic images for the
following reasons:
- dd doesn't preserve sparseness
- truncate syntax is much more clear
- dd requires additional calculations of the image size
in blocks
- the way dd was used in the code is not always correct.
In some cases it was writing one block to the file which makes
it not 100% sparse.
[YOCTO #9099]
(From OE-Core rev: d2d0d18dfd3922411d856b98ab6ba5d64c9c1c9f)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Added explanation of --system-id option to the output of
wic help kickstart.
[YOCTO #9096]
(From OE-Core rev: 1a304afea4ad7be12ed5f0fcb397a538345a6b63)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Used sfdisk to set partition system id if --system-id parameter
is used for a partition in wks file.
[YOCTO #9096]
(From OE-Core rev: a1f7f7e61fd20fb6319825648930f7b6aa0e0cee)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Added new argument to add_partition call to pass partition
system id down the stack.
[YOCTO #9096]
(From OE-Core rev: f2733df697192c0010c17b7bbb02f8679cb8f313)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Added Partition.system_id attribute and initialized it
from parse result of wks option --system-id. It will be
used by the wic code below the call stack to set partition
system id.
[YOCTO #9096]
(From OE-Core rev: 4f195a5b7574ebff8fbdb3045daa71f173f97a30)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Added sfdisk -> util-linux pair to the dictionary
executable -> recipe as sfdisk is going to be used by wic
to set partition system id.
[YOCTO #9096]
(From OE-Core rev: 398aafa185acbc7239505f7107735e93a502f6d2)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Added new option --system-id to wks parser. The option
will be used to set partition system id.
[YOCTO #9096]
(From OE-Core rev: b9c56b1c95cd1d0fd809d257e0cd05a50c481bed)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
os.path.join discards the cr_workdir var contents if the path of the
second arguments is absolute.
(From OE-Core rev: dba099d77dcc66b239523a55f3ed26784f9a662a)
Signed-off-by: Ioan-Adrian Ratiu <adrian.ratiu@ni.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The latest wic kickstart refactoring introduced a bootloader option
"--configfile" which lets wks' specify a custom grub.cfg for use
while booting. This is very useful for creating stuff like boot menus.
This change lets isoimage-isohybrid use --configfile; if this option is
not specified in a wks, it generates a default cfg as before.
(From OE-Core rev: bf673a769514b13558ad9c785ae4da3a5adfd1e0)
Signed-off-by: Ioan-Adrian Ratiu <adrian.ratiu@ni.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If UNINATIVE_CHECKSUM changes over an SDK update, bitbake within the
extensible SDK will be broken because it will see that the matching
uninative tarball doesn't exist and if there is a default value of
UNINATIVE_URL it will attempt to download the file and will then fail
because the checksums don't match up; alternatively if no UNINATIVE_URL
is set then it'll also fail with an error about misconfiguration. To fix
this, add some logic to devtool sdk-update to download the matching
uninative tarball(s) for the checksum(s) in the newly fetched SDK
configuration.
Fixes [YOCTO #9301].
(From OE-Core rev: 14ff58ad98a5afac08db77068d80f152d8875766)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Replace git pull with fetch and reset to avoid the merge logic in the
event that the layers repo in the published SDK we're updating to isn't
fast-forward merge from the local repo.
Also add gitignore and committer info during publish to avoid errors and
to be sure that the first commit has a dummy user in it.
[ YOCTO #9368 ]
(From OE-Core rev: 4657bc9d165e51981e034e73e7b92552e873eef7)
Signed-off-by: Stephano Cetola <stephano.cetola@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is unusual but not impossible to find recipes whose first entry is
not the main source URL but instead some patch or other local file, for
example python-cryptography in meta-python (which sets SRC_URI before
inheriting pypi). There's nothing inherently wrong with this, and we
shouldn't assume that the first entry is the main source URL, so just
take the first non-local entry instead.
(From OE-Core rev: c868198c1f6006789707b497c2ae34d7cc5e706f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For example, support both:
$ runqemu qemux86-64 ramfs
$ runqemu qemux86-64 cpio.gz (new)
(From OE-Core rev: 6529264776701d4f5a1e4a8336ac2e01a6ddea85)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Partitions specifying --fsoptions were silently skipped by wic
due to the old bug introduced when removing code related to
subvolume handling:
- if mountpoint == "/" or not fsopts or fsopts.find("subvol=") == -1
+ if mountpoint == "/" or not fsopts:
[YOCTO #9396]
(From OE-Core rev: be7ff1741e8ab5f2724b3f64da1bed8b0d3dcb7c)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ed Bartosh
Patrick Ohly
Randy Witt
Leonardo Sandoval
Christopher Larson
Paul Eggleton
Ioan-Adrian Ratiu
Peter Kjellerstedt
Stephano Cetola
Robert Yang