Commit Graph

76 Commits

Author SHA1 Message Date
Olaf Mandel 9af59d07a3 combo-layer: handle file_exclude matching dirs
If file_exclude matches a directory, os.unlink() got called with this
directory as an argument.

Filter out paths that end in a directory separator.

This still leaves the (then empty) directories, but this does not affect
the git commit.

(From OE-Core rev: e84b9185cc8f8e9f9df0e050543bb3a2c59426c3)

Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-28 16:15:21 +01:00
Olaf Mandel e02f0f1cb1 combo-layer: handle ambiguous git arguments
If a branch/src-repository has the same name as a file/directory, git
since 1.4.0(?) gives an error like the one below:
ambiguous argument 'bitbake': both revision and filename
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

Add two dashes to make the intent clear.

(From OE-Core rev: a84ec3ac15a59f72fcb46d97942009c8e459b5d0)

Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-28 16:15:18 +01:00
Olaf Mandel 050b8969ab combo-layer: fix crashes on wrong tempfile usage
When calling tempfile.NamedTemporaryFile().write(str()), at least on
Python 3.4.2 this fails with this error:
TypeError: 'str' does not support the buffer interface

Change the file-mode for all such files from binary to text mode.

(From OE-Core rev: d08f3882a35eec8a042d2501715684444e353605)

Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-28 16:15:18 +01:00
Ed Bartosh a81b326933 combo-layer: python3: fix UnicodeDecodeError
check_patch function opens patch file in text mode. This causes
python3 to throw exception when calling readline():
    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa7 in position
                        NNNN: invalid start byte

Opening file in binary mode and using binary type instead of strings
should fix this.

(From OE-Core rev: a7f1435c4c26237cdb55066c9f5408b4fdf016aa)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-24 13:58:28 +01:00
Ed Bartosh cdff6bc0c1 scripts: python3: change python to python3 in shebang
(From OE-Core rev: 4b544ff388497cac82b0585f237900595523e1cb)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-03 13:13:29 +01:00
Ed Bartosh 4e36eeee77 scripts: python3: decode subprocess output
stdeout and stderr content returned by subprocess API has different
types in Python 3(bytes) and Python 2(string). Decoding it to
'utf-8' makes it unicode on both pythons.

(From OE-Core rev: ae4d36375683b6cfd48af25bfca70b2e77f7ade5)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-03 13:13:29 +01:00
Ed Bartosh a8314b9531 combo-layer: python3: use tempfile.TemporaryFile
Used tempfile.TemporaryFile() API instead of deprecated
os.tmpfile().

(From OE-Core rev: bf1b411eb1cd2cc960325d5fdb0cb4f4f7b1e40e)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-03 13:13:29 +01:00
Ed Bartosh a4045424af combo-layer: python3: import reduce
Reduce is not a builtin function in python3.
It has to be imported from functools.

(From OE-Core rev: f9bda8b36decb5a837adcfeb1173092401988801)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-03 13:13:29 +01:00
Ed Bartosh ee31bad762 scripts: python3: use new style except statement
Changed old syle except statements 'except <exception>, var'
to new style 'except <exception> as var' as old style is not
supported in python3.

(From OE-Core rev: 438eabc248f272e3d272aecaa4c9cec177b172d5)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-03 13:13:28 +01:00
Ed Bartosh 07c97db272 scripts: python3: convert iterables to lists
Converted return value of items() keys() and values() to
lists when dictionary is modified in the loop and when
the result is added to the list.

(From OE-Core rev: 874a269eb1d70060c2f3b3f8b70800e2aea789f4)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-03 13:13:28 +01:00
Ed Bartosh fa4275bbc0 scripts: Rename ConfigParser -> configparser for python3
The ConfigParser API was renamed to configparser in python 3.
Renamed ConfigParser -> configparser in scripts/ to make the
code working in python 3.

(From OE-Core rev: de6e98f272e623ce72e724e66920eecf10cb2d41)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:01 +01:00
Ed Bartosh 7eab022d4b scripts: Fix deprecated dict methods for python3
Replaced iteritems -> items, itervalues -> values,
iterkeys -> keys or 'in'

(From OE-Core rev: 25d4d8274bac696a484f83d7f3ada778cf95f4d0)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:01 +01:00
Ed Bartosh 2e388048b6 scripts: python3: Use print function
Used print function instead of print statement to make
the code work in python 3.

(From OE-Core rev: 80fecc44761fa38ccf2e4dc6897b9f1f0c9c1ed0)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-01 12:47:10 +01:00
Patrick Ohly 4dadbbdd46 combo-layer: avoid too long command lines in update with history
As suspected, invoking "git archive" with all intended files as
parameters can run into command line length limitations. Splitting up
the parameters into multiple invocations (xargs-style) works and was
tested after encountering the situation in practice.

(From OE-Core rev: 1cb484ab99eabb5c24792757ab09d7f170f2e614)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-14 23:05:13 +01:00
Patrick Ohly 659ed8718b combo-layer: fix default "update" mode
When the "history" option is not set in the combo-layer.conf, the
intended default was to use the traditional method. Passing "True" as
default when querying the config was unintentional.

Also remove some left-over debugging code.

(From OE-Core rev: d0304acb05b926b08805d8652e12eaf19bf53ad6)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-14 23:05:12 +01:00
Patrick Ohly d53ed05390 combo-layer: implement "update with history"
The core idea is that all commits get imported, including merge
commits, and joined into one big merge commit that imports the changes
from the individual components into the main branch of the combined
repository.

This is done by copying the files in each commit and removing deleted
ones, instead of trying to patch the combined repository.

The advantages of doing updates in this mode are:
- works for arbitrary upstream repos, not just those which
  support conversion into a linear set of patches
- listing history shows that commits where developed
  independently in the different components, instead of
  artificially showing them as if they had been developed
  one after the after (component "aaaa" before "bbbb", then "ccc", ...)
- bisecting becomes easier: when upstream repos only ensure consistency
  when merging into their "master" branches, then those merge
  commits are good candidates for test builds also in the combined
  repo
- more data mining can be done, for example showing who merged a commit
  and when

Selecting a subset of the files is supported, albeit with a slight
different semantic for wild card matching compared to other code paths
(/ is matched by * and ?). Empty commits get skipped because typically
they are a result of filtering (but that is not checked, so
intentionally empty commits also get skipped).

Other code paths are intentionally left unchanged, to avoid
regressions. However, the downside is that some opportunities for
refactoring (in particular regarding file filtering) were ignored.

(From OE-Core rev: 660f76b6fb0fb95738a2c8f50e0a99ffa5831c64)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-11 10:33:41 +01:00
Patrick Ohly 107c35e87e combo-layer: runcmd() enhancements
Allow setting the environment. Due to a subprocess quirk, it must
always be set explicitly (reuses the one from the previous call if not
set, instead of falling back to os.environ).

Embedding nul characters will be useful for parsing git output more
reliably; support dumping such output a bit better.

(From OE-Core rev: 0af4fadafce690fc8357196cf7247bd222c08d10)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-11 10:33:41 +01:00
Patrick Ohly aa4de3c8c5 combo-layer: dummy "update with history"
When setting "history = True" in combo-layer.conf consistently for the
components involved in an update or using "update" together with the
"--history" command line flag, a new mode for updating will be used
that does not rely on exporting/importing patches.

A config setting is used because it should be used consistently by
everyone using the same config, without having to remember to use an
additional command line parameter.

There are no real global settings, so the setting is checked
separately for each component although the setting has to be set
consistently. This restriction could be removed later.

In practice, putting "history" into the "[DEFAULT]" section is the
easiest approach for configuring it.

The actual code changes split up action_update and the
combo-layer.conf handling in preparation for this new mode, without
implementing the mode itself.

(From OE-Core rev: c9dab31f5f6dc225f5c2c2ca3ec9aeab2ff655d5)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-11 10:33:41 +01:00
Richard Purdie c37d5426b1 scripts, lib: Don't limit traceback lengths to arbitrary values
There appears to have been a lot of copy and pasting of the code
which prints tracebacks upon failure and limits the stack trace to
5 entries. This obscures the real error and is very confusing to the user
it look me an age to work out why some tracebacks weren't useful.

This patch removes the limit, making tracebacks much more useful for
debugging.

[YOCTO #9230]

(From OE-Core rev: 6069175e9bb97ace100bb5e99b6104d33163a3a2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-31 23:01:36 +01:00
Patrick Ohly 431c1e171f combo-layer: handle empty commits during "init --history"
When importing the components during the "combo-layer init" with full
history and relocation into a destination directory, components with
empty commits were not handled because the "mv" command was invoked
with just one parameter.

Replacing that with a find/xargs pair avoids the problem and should
also fix the handling of filenames with special characters (in
particular spaces).

(From OE-Core rev: fd9d08b8ba512151615a03fda61a103760a0683c)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-02 22:39:43 +00:00
Richard Purdie e6f27619af combo-layer: Stop using filterdiff
I ran into an issue where a patch just deleting a single file
within the repository (meta/recipes-devtools/m4/m4/make.patch)
would get skipped by combo-layer.

It turns out this has the patch header (commented to avoid breaking scripts):

: diff --git a/meta/recipes-devtools/m4/m4/make.patch b/meta/recipes-devtools/m4/m4/make.patch
: deleted file mode 100644
: index 79fb415..0000000
: --- a/meta/recipes-devtools/m4/m4/make.patch
: +++ /dev/null
: @@ -1,42 +0,0 @@

and this is classed as > 5 headers in filterdiff. When we piped the path
through filterdiff, the --- line disappears, then the second time we pass
through filterdiff, it shows no lines changed and the patch is assumed
to be empty and skipped.

Changing MAX_HEADERS in filterdiff is one way to fix this, another would
be to grep out "deleted file mode" lines. Instead, we can use new
git syntax to exclude files from the git format-patch instead and avoid
filterdiff entirely.

(From OE-Core rev: 296c70afeef75396dea9ae436058314d406dc257)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 12:18:18 +00:00
Alexander D. Kanevskiy aa1844e847 combo-layer: introduce ability to exclude component from mass update
There is no ability at the moment for situations where users would like
to keep section in combo-layer.conf but don't update it, unless explicitly
specified.

Now, by adding "update = no" to desired section would exclude
that repository from "combo-layer update" command.

It is still possible to explicitly update it by "combo-layer update $section".

By default, all repositories are assumed as "update = yes"

(From OE-Core rev: 74ecf1a0e5c8230e55cb1a309f711d98fa63ce79)

Signed-off-by: Alexander D. Kanevskiy <kad@kad.name>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-11-16 11:39:31 +00:00
Patrick Ohly 8b847ffd04 combo-layer: fix action_pull for unknown branch
When reconfiguring the branch to something not already fetched,
action_pull fails with
   error: pathspec '<new branch name>' did not match any file(s) known to git.

It is the "git checkout" which fails like that. To solve this,
try the faster "git checkout + git pull" first and only if that fails,
fall back to the slow "git fetch + git checkout".

In the conf.hard_reset case, do the checkout always after the git fetch.

(From OE-Core rev: ad4d3b1da190cf08c6ac5f9a94a2a1c4980a184d)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-09 00:13:57 -07:00
Patrick Ohly b2a0c2fbd1 combo-layer: enhance output in action_pull
The git operations can fail, for example when the branch is unknown
or misconfigured.

Better move the info message and extend it such that it is printed
first and provides the necessary context, because otherwise the
CalledProcessError exception gets dumped without mentioning for which
component it occurred.

(From OE-Core rev: 00d01468a692faf4272894dd328e8c532bcf8b49)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-09 00:13:57 -07:00
Alexander Kanevskiy 405bb5b588 combo-layer: fix AttributeError traceback.
Commit c908a423f85a84ddd8249abd00254f29d47df74b introduced a new
issue in combo-layer that leads to a traceback as
args.hard_reset is an unknown variable. This change defines an
appropriate destination for the command args parser and fixes the
reference.

(From OE-Core rev: ac4ef551d506a5767946cf2fb717c2fd7f86df9c)

Signed-off-by: Alexander Kanevskiy <kad@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-07-31 12:09:42 +01:00
Markus Lehtonen b0079ec5e9 combo-layer: implement --hard-reset option
This option causes combo-layer to do git fetch and hard reset instead of
git pull in the component repositories. This makes sure that the local
component repositories are always in sync with the remote - tolerating
force pushes and overriding any locally made changes.

(From OE-Core rev: c908a423f85a84ddd8249abd00254f29d47df74b)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-07-16 15:09:17 +01:00
Markus Lehtonen d1a1dbb8a5 combo-layer: only allow fast-forward when pulling
Only allow fast-forward merges in the component repositories when doing
git-pull. This makes it possible to spot problems (i.e. rewriting of
history) in the component upstream . Also, this change prevents the
creation of local-only merge commits in the component repositories.
These merges cause "last_revision" field of the combo-layer config to
point to a git commit that is only present in the users local component
repository but nowhere in upstream.

(From OE-Core rev: 5207169b9c6c4b05c6b043745905a5498adca1a3)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-07-16 15:09:17 +01:00
Puustinen, Ismo daec53e977 combo-layer: modified to generate better commit messages.
This patch includes support for a global section in combo-layer.conf
called [combo-layer-settings]. Supported in this section is key
"commit_msg"; its value is the template for the git commit message
that updates the last_revision. The template can include substitution
for the updated component list: ${components}. The substituted value
will either be a comma-separated list of components or "all components",
if combo-layer was invoked without component list argument.

If the key is not present, the old default value is used for the commit
message.

Configuration file example:

[combo-layer-settings]
commit_msg = pulled in the latest changes for ${components}.

(From OE-Core rev: fe84747f961772b61031af59d44e54b178148379)

Signed-off-by: Ismo Puustinen <ismo.puustinen@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-07-08 00:01:22 +01:00
Richard Purdie 20a3a36547 scripts/combo-layer: Fix exit codes and tty handling
If combo-layer is called from a non-interactive context we should exit
with a correct error code rather than try and drop to a shell.

This patch cleans up a few error case exit codes as well as
detecting and handling non-interactive usage.

(From OE-Core rev: 3b1d89a51445cf526ca84eb5b53de434f9585d6e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-06-05 16:42:17 +01:00
Patrick Ohly 45b8e62439 combo-layer: handle unset dest_dir in sanity_check()
The previous "clean up dest_dir checking" patch (f8cdbe7497) improved
handling of empty dest_dir but made handling of unset dest_dir worse:
instead showing the "Option dest_dir is not defined for component ..."
error, it fails with a Python exception.

Avoid that by providing a sane fallback for the unset case. With that
change, dest_dir is no longer strictly required, but the check for it
is kept to ensure that a combo-layer.conf also works with older
combo-layer versions.

[Yocto #7773]

(From OE-Core rev: d4bf858b2c15bef128fd6d606b08203a318e2d4c)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-05-22 13:29:47 +01:00
Patrick Ohly dcb8f89723 combo-layer: improve merge commit handling
When the head of a branch is a merge commit, combo-layer did not
record that commit as last_revision because it only considers applied
patches, and the merge commit never gets applied.

This causes problems when the merge commit leads to multiple patches
and the commit id that gets recorded only reaches some of these
patches. The next run then will try to re-apply the other patches.

This special case is now detected and dealt with by bumping
last_revision to the branch commit. The behavior where the head is a
normal commit is intentionally not changed, because some users might
prefer the traditional behavior.

(From OE-Core rev: 701bcd52c208f22a9a6c48a11a35bcf1c4e413df)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-05-09 22:26:45 +01:00
Patrick Ohly 240dadea12 combo-layer: fix file_exclude for empty commits
The code detecting empty patches after removing files with
file_exclude failed for commits which were already empty before (like
the initial commit in some repos): such patches are completely empty
files, without a From line.

Detect that case and just let the normal empty patch detection deal
with it.

(From OE-Core rev: 3e310b8cd603539a2de115b2b73e1db35403b426)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-03-31 22:23:17 +01:00
Patrick Ohly 3775c6e714 combo-layer: fix file_exclude for dest_dir = .
"filterdiff -x ./some/file" does not remove changes for some/file.
We must be more careful about constructing the path name and
only add the prefix when it really means a directory.

While at it, also better normalize the path in copy_selected_files()
early on, to handle double slashes. Useful should the function ever
gets used for something other that dest_dir (which gets normalized in
sanity_check()).

(From OE-Core rev: 8ea63c6c920c39e5a4ba176223fe472f92e2632a)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-03-31 22:23:16 +01:00
Patrick Ohly 5508950045 combo-layer: clean up dest_dir checking
Empty dest_dir is basically undocumented behavior. The sample conf
only mentions using just a dot for the current directory. In practice,
the empty string does not work because of code like this:

def action_splitpatch(conf, args):
    ...
        if dest_dir != ".":
            filerange_root = '%s -x "%s/*"' % (filerange_root, dest_dir)

However, the empty string was not explicitly checked for, leading to
strange errors when trying to apply patches:

[12:50:23] Applying: foobar: xyz
fatal: unable to stat newly created file '/foobar': No such file or directory

This patch turns the empty string into an alias for the dot. This seems
more user-friendly than throwing an error. This alias is intentionally
not document in the sample conf, because the dot is clearer and works also
with older copies of combo-layer.

Instead of checking for both all the time and normalizing the path when
needed (as done in some places), rewrite the value in sanity_check()
and then only check for '.'.

(From OE-Core rev: f8cdbe749755dc769150d3a6c2c54318c80e1562)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-03-31 22:23:16 +01:00
Richard Purdie c5e9654ac9 scripts/combo-layer: Handle update with no repo/revision specified
Running an update operation with no repo/revision specified was failing.
This fixes that code path which worked until the change from:

http://git.yoctoproject.org/cgit.cgi/poky/commit/scripts/combo-layer?id=3592507a149b668c0a925e176535f7f2002fa543

(From OE-Core rev: 411a6a3694a9fcd563c5b5320597314fbec75cc2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-03-20 23:56:06 +00:00
Patrick Ohly e0d3cabc5a combo-layer: partial import for '--history init'
The new "since_revision" property can be used to cut off the imported
history at some point. This is useful to keep the resulting repository
smaller while still preserving enough history that "git annotate"
reports the right author and commit for most lines.

The initial, squashed import commit shows up with "unknown" as author
in the "git annotate" output. It has the repository name as prefix
in the subject line; importing that commit works best with a
layer hook which does not add the repository name again when
it is already present. Adding it here is useful for hooks
which do not extend the subject line.

(From OE-Core rev: 74f4c9e3bcdb3c4ca919623086e92a9379bd81ff)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-03-20 11:21:24 +00:00
Patrick Ohly 14eac2a474 combo-layer: combine trees via replacement objects
Instead of local graft entries rooting the imported branches in the
shared root commit, use replacement objects.

The advantage is that they get moved around by "git push" and "git
fetch", so everyone has the same, nicer view with everything starting
at the beginning of the combined repository.

If undesired, these objects can be removed with "git replace".

(From OE-Core rev: 220d816cda3c9b8d888288cc03eb74be5e71cc59)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-03-20 11:21:24 +00:00
Patrick Ohly dd985a241c combo-layer: init with full history
The new --history parameter enables a new mode in "combo-layer init"
where it copies the entire history of the components into the new
combined repository. This also imports merge commits.

Moving into a destination directory and applying commit hooks
is done via "git filter-branch" of the upstream branch. File
filtering uses the same code as before and just applies it
to that filtered branch to create the final commit which
then gets merged into the master branch of the new repository.

When multiple components are involved, they all get merged
into a single commit with an octopus merge. This depends
on a common ancestor, which is grafted onto the filtered
branches via .git/info/grafts.

These grafts are currently left in place. However, they do not get
pushed, so the local view on the entire history (all branches rooted
in the initial, empty commit, temporarily diverging and then
converging) is not the same as what others will see (branches starting
independently and converging). Perhaps "git replace" should be used
instead.

The final commit needs to be done manually, as before. A commit
message with some tracking information is ready for use as-is. This
information should be sufficient to implement also "combo-layer
update" using this approach, if desired. The advantage would be that
merge commits with conflict resolution would not longer break
the update.

(From OE-Core rev: 9e40cb1ab77029df7f2cf1e548a645ff6a62c919)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-03-20 11:21:23 +00:00
Patrick Ohly b4326bf85a combo-layer: update() also instance property
The Configuration class mirrors all properties in local hashes.
When updating the configuration, also update these properties
to remain consistent.

(From OE-Core rev: c575ff183ae5c35e0f7e8d90c222994574ce73c9)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-03-20 11:21:23 +00:00
Patrick Ohly 96b98451c4 combo-layer: exclude files
Some combined repos intentionally do not include certain files.
For example, Poky does not include bitbake's setup files and
OE-core's sample files under meta/conf.

When these files get modified in the upstream repository, applying the
patches fails and requires manual intervention. That is merely a
nuisance for someone familiar with the problem, but a real show
stopper when having the import run automatically or by someone less
experienced.

Therefore this change introduces "file_exclude", a new per-repo list
of file patterns which removes all matching files when initializing or
updating a combined repository. Because fnmatch is used under the hood
to match full path strings, removing entire directories must be done
with a pattern ending in a '/*' (in contrast to file_filter).

For Poky, the additional configuration looks like this:

[bitbake]
...
file_exclude = classes/base.bbclass
	conf/bitbake.conf
	.gitignore
	MANIFEST.in
	setup.py
	TODO

[openembedded-core]
...
file_exclude = meta/conf/bblayers.conf.sample
	meta/conf/local.conf.sample
	meta/conf/local.conf.sample.extended
	meta/conf/site.conf.sample

(From OE-Core rev: a51b37f4db6b144386d1bd5789ec91acc78a0bd8)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-03-20 11:21:23 +00:00
Patrick Ohly 627e524238 combo-layer: runcmd() with separate output
Allow the caller to specify a separate output stream. stderr is always
a temporary file opened by runcmd(), so read from that to capture
output for error reporting *and* the return value.

The reasoning for the latter is a) that this preserves the traditional
behavior when out=None and b) if the caller wants the content of
stdout, it can read from the stream itself, which is not possible for
the temporary stderr.

(From OE-Core rev: a084162a9dc4718ab453723f1f28aefc55100e2e)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-03-20 11:21:22 +00:00
Patrick Ohly 58c5bb28b5 combo-layer: make Signed-off-by optional
It depends on the diligence of the person running the combo-layer tool
whether the Signed-off-by line added to each commit actually indicates
that the person was involved in validating the change.

When the import is purely automatic, it is better to not add the line,
because the history is more useful without it (searching for the person
really only lists changes he or she was involved with) and it would
be a false statement.

The 'signoff' property can be set per repository, like every
other property. But setting it in the special [DEFAULT] section
is more useful, so that is what the example shows.

(From OE-Core rev: dba34e5a6f0a4c726266654dc5f26e0f7f13db0f)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-03-20 11:21:22 +00:00
Patrick Ohly b1408ba706 combo-layer: let user choose where properties get updated
When updating a property (at the moment, only 'last_revision'),
first check whether the user already added it to the main config.
If so, update there even if there is a local config.

This way, 'last_revision' can be shared between developers
as part of the repository while still configuring per-developer
repo paths outside of the git repository in a local config.

An example of a repository which is set up like that is tizen-distro,
with instructions for such a collaborative maintenance found here:
https://review.tizen.org/git/?p=scm/bb/tizen-distro.git;a=blob;f=README;hb=refs/heads/tizen-ivi

(From OE-Core rev: a3b246ee928d2ec59c8b6a775d9309f5be33ecc7)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-03-20 11:21:21 +00:00
Markus Lehtonen 3592507a14 combo-layer: support updating up to arbitrary commit
Support defining the top commit up to which to update. In other words,
this makes it possible to update up to certain point other than the
branch head. The update point (git commitish) is given on the command
line by appending the component name(s) with a colon and the commitish,
e.g.
 $ combo-layer update my_component:sha1

Only the "update" action supports this.

(From OE-Core rev: b7ee7cf5bbfd0f305a75e4df9455f9968aa67878)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-01-29 15:36:52 +00:00
Markus Lehtonen 62d94a44ae combo-layer: minor refactor
Change get_repos() to assume a list of repository names instead of full
list of command line arguments.

(From OE-Core rev: d2d8f3c2a7570bb74db713ddc83059d3b3bd4b2e)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-01-29 15:36:51 +00:00
João Henrique Ferreira de Freitas 1ef8100d59 combo-layer: fix revlist taking into account file_filter
If file_filter is set, git format-patch takes account but git rev-list
does not. So revlist is going to get with wrong revisions. And last_revision
will be updated with wrong revision. The next time that user run
combo-layer it complain about applying patches.

So ensure that 'git rev-list' are using file_filter as 'git format-patch'.

(From OE-Core rev: c22e40900d8e33c2c884e714c11ddb771b86923f)

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-06-01 14:29:30 +01:00
Saul Wold 51cccb7fb9 combo-layer: Make directories only if they don't exisit yet
(From OE-Core rev: 24188967209bad291545909ddb89af35ab3f6021)

Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-01-28 00:52:37 +00:00
Richard Purdie 214ad32077 scipts/combo-layer: Fix check_rev_branch() for cases where the revision is on more than one branch
If a revision is in more than one branch, the check_rev_branch() function can't
cope with it and the tool returns incorrect errror messages. This patch
ensures it copes with this situation.

(From OE-Core rev: 14bd101c6a86dd048da98817f47694fb21504209)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-10-18 12:13:42 +01:00
Paul Eggleton a2eab6bb16 scripts/combo-layer: ensure we validate branch/revision on init
If both branch and last_revision are specified for a component when
combo-layer init is run, ensure that the specified revision is actually
on the specified branch and error out if not. Also ensure that the error
message mentions the component.

(From OE-Core rev: e498257ecbec94cec181d73bda57d44335b4dee0)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-09-14 09:50:29 +01:00
Paul Eggleton 7ddf64d58d scripts/combo-layer: use last_revision if specified in init
If last_revision is specified for a component when running combo-layer
init, then use that revision instead of the latest revision on the
branch. Also, remove unnecessary git checkout during init since we
specify the revision to all calls to git when dealing with the component
repositories.

Fixes [YOCTO #3040].

(From OE-Core rev: ff8277cd133e9a02b131977078cff61fa587a1af)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-09-14 09:50:28 +01:00