The new "since_revision" property can be used to cut off the imported
history at some point. This is useful to keep the resulting repository
smaller while still preserving enough history that "git annotate"
reports the right author and commit for most lines.
The initial, squashed import commit shows up with "unknown" as author
in the "git annotate" output. It has the repository name as prefix
in the subject line; importing that commit works best with a
layer hook which does not add the repository name again when
it is already present. Adding it here is useful for hooks
which do not extend the subject line.
(From OE-Core rev: 74f4c9e3bcdb3c4ca919623086e92a9379bd81ff)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of local graft entries rooting the imported branches in the
shared root commit, use replacement objects.
The advantage is that they get moved around by "git push" and "git
fetch", so everyone has the same, nicer view with everything starting
at the beginning of the combined repository.
If undesired, these objects can be removed with "git replace".
(From OE-Core rev: 220d816cda3c9b8d888288cc03eb74be5e71cc59)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The new --history parameter enables a new mode in "combo-layer init"
where it copies the entire history of the components into the new
combined repository. This also imports merge commits.
Moving into a destination directory and applying commit hooks
is done via "git filter-branch" of the upstream branch. File
filtering uses the same code as before and just applies it
to that filtered branch to create the final commit which
then gets merged into the master branch of the new repository.
When multiple components are involved, they all get merged
into a single commit with an octopus merge. This depends
on a common ancestor, which is grafted onto the filtered
branches via .git/info/grafts.
These grafts are currently left in place. However, they do not get
pushed, so the local view on the entire history (all branches rooted
in the initial, empty commit, temporarily diverging and then
converging) is not the same as what others will see (branches starting
independently and converging). Perhaps "git replace" should be used
instead.
The final commit needs to be done manually, as before. A commit
message with some tracking information is ready for use as-is. This
information should be sufficient to implement also "combo-layer
update" using this approach, if desired. The advantage would be that
merge commits with conflict resolution would not longer break
the update.
(From OE-Core rev: 9e40cb1ab77029df7f2cf1e548a645ff6a62c919)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The Configuration class mirrors all properties in local hashes.
When updating the configuration, also update these properties
to remain consistent.
(From OE-Core rev: c575ff183ae5c35e0f7e8d90c222994574ce73c9)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some combined repos intentionally do not include certain files.
For example, Poky does not include bitbake's setup files and
OE-core's sample files under meta/conf.
When these files get modified in the upstream repository, applying the
patches fails and requires manual intervention. That is merely a
nuisance for someone familiar with the problem, but a real show
stopper when having the import run automatically or by someone less
experienced.
Therefore this change introduces "file_exclude", a new per-repo list
of file patterns which removes all matching files when initializing or
updating a combined repository. Because fnmatch is used under the hood
to match full path strings, removing entire directories must be done
with a pattern ending in a '/*' (in contrast to file_filter).
For Poky, the additional configuration looks like this:
[bitbake]
...
file_exclude = classes/base.bbclass
conf/bitbake.conf
.gitignore
MANIFEST.in
setup.py
TODO
[openembedded-core]
...
file_exclude = meta/conf/bblayers.conf.sample
meta/conf/local.conf.sample
meta/conf/local.conf.sample.extended
meta/conf/site.conf.sample
(From OE-Core rev: a51b37f4db6b144386d1bd5789ec91acc78a0bd8)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow the caller to specify a separate output stream. stderr is always
a temporary file opened by runcmd(), so read from that to capture
output for error reporting *and* the return value.
The reasoning for the latter is a) that this preserves the traditional
behavior when out=None and b) if the caller wants the content of
stdout, it can read from the stream itself, which is not possible for
the temporary stderr.
(From OE-Core rev: a084162a9dc4718ab453723f1f28aefc55100e2e)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It depends on the diligence of the person running the combo-layer tool
whether the Signed-off-by line added to each commit actually indicates
that the person was involved in validating the change.
When the import is purely automatic, it is better to not add the line,
because the history is more useful without it (searching for the person
really only lists changes he or she was involved with) and it would
be a false statement.
The 'signoff' property can be set per repository, like every
other property. But setting it in the special [DEFAULT] section
is more useful, so that is what the example shows.
(From OE-Core rev: dba34e5a6f0a4c726266654dc5f26e0f7f13db0f)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When updating a property (at the moment, only 'last_revision'),
first check whether the user already added it to the main config.
If so, update there even if there is a local config.
This way, 'last_revision' can be shared between developers
as part of the repository while still configuring per-developer
repo paths outside of the git repository in a local config.
An example of a repository which is set up like that is tizen-distro,
with instructions for such a collaborative maintenance found here:
https://review.tizen.org/git/?p=scm/bb/tizen-distro.git;a=blob;f=README;hb=refs/heads/tizen-ivi
(From OE-Core rev: a3b246ee928d2ec59c8b6a775d9309f5be33ecc7)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Support defining the top commit up to which to update. In other words,
this makes it possible to update up to certain point other than the
branch head. The update point (git commitish) is given on the command
line by appending the component name(s) with a colon and the commitish,
e.g.
$ combo-layer update my_component:sha1
Only the "update" action supports this.
(From OE-Core rev: b7ee7cf5bbfd0f305a75e4df9455f9968aa67878)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Change get_repos() to assume a list of repository names instead of full
list of command line arguments.
(From OE-Core rev: d2d8f3c2a7570bb74db713ddc83059d3b3bd4b2e)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If file_filter is set, git format-patch takes account but git rev-list
does not. So revlist is going to get with wrong revisions. And last_revision
will be updated with wrong revision. The next time that user run
combo-layer it complain about applying patches.
So ensure that 'git rev-list' are using file_filter as 'git format-patch'.
(From OE-Core rev: c22e40900d8e33c2c884e714c11ddb771b86923f)
Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a revision is in more than one branch, the check_rev_branch() function can't
cope with it and the tool returns incorrect errror messages. This patch
ensures it copes with this situation.
(From OE-Core rev: 14bd101c6a86dd048da98817f47694fb21504209)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If both branch and last_revision are specified for a component when
combo-layer init is run, ensure that the specified revision is actually
on the specified branch and error out if not. Also ensure that the error
message mentions the component.
(From OE-Core rev: e498257ecbec94cec181d73bda57d44335b4dee0)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If last_revision is specified for a component when running combo-layer
init, then use that revision instead of the latest revision on the
branch. Also, remove unnecessary git checkout during init since we
specify the revision to all calls to git when dealing with the component
repositories.
Fixes [YOCTO #3040].
(From OE-Core rev: ff8277cd133e9a02b131977078cff61fa587a1af)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Handle the case during update where the configured branch is not
currently checked out in the component repository by just specifying
it in the places where it was not previously.
(From OE-Core rev: a3ddf39af7e7cd47156677d8ae753964ee582745)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the local configuration is already split out, ensure the init action
works properly and does not error in the case that last_revision is not
set. Additionally, if the local configuration is within the repository,
prevent it from being committed and add it to .gitignore.
(From OE-Core rev: de339b0cb201035e27df1128ccf526937b8325ec)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow splitting the local parts of the configuration (mostly
local_repo_dir and last_revision, although there is no limitation) to
a side-by-side -local.conf file, with component sections optionally
tagged with the combo layer branch name. This effectively allows you to:
* avoid polluting the history by committing the updated last revision
to the combo repository for every update
* avoid putting local repo paths into the combo repository
* manage multiple branches of the combo repository whilst avoiding the
possibility of mixing the configuration for one branch with another.
An example split configuration (note, values may be artificial):
------------------- combo-layer.conf -------------------
[bitbake]
src_uri = git://git.openembedded.org/bitbake
dest_dir = bitbake
hook = scripts/combo-layer-hook-default.sh
[oe-core]
src_uri = git://git.openembedded.org/openembedded-core
dest_dir = .
hook = scripts/combo-layer-hook-default.sh
--------------------------------------------------------
---------------- combo-layer-local.conf ----------------
[bitbake]
local_repo_dir = ../repos/bitbake
[oe-core]
local_repo_dir = ../repos/oe-core
[bitbake|master]
branch = master
last_revision = db689a99beffea1a285cdfc74a58fe73f1666987
[oe-core|master]
branch = master
last_revision = 121a1499a81706366acc0081272a6bff634d4d62
[bitbake|denzil]
branch = 1.12
last_revision = 24b631acdaa143a4de39c6e1328849660c66f219
[oe-core|denzil]
branch = denzil
last_revision = 741146fa90f28f7ce8d82ee7f7e254872d519724
--------------------------------------------------------
It is assumed that the local config file will be added to .gitignore.
(From OE-Core rev: f0065d7a6973628803a17c57f2265512aba3234c)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the user edits the configuration file by hand and sets last_revision,
we need to ensure that the revision is valid and on the specified
branch.
(From OE-Core rev: 05382932257257247b8c18bc06e9c0039d134d06)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Ignore blank lines in patch list
* Don't fail in interactive mode if patch list is deleted
* Show patch counter
* Show relative path for patches
* Print headings before applying patch list for each component
Also change to using a "with" block to read the patch list so it gets
closed properly when we're finished.
Fixes [YOCTO #2455].
(From OE-Core rev: 65461d7c35fdadb5b008052798731dce19ed187f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If applying a patch fails during the update process, drop to a shell
instead of exiting; at that point the user can manually apply the patch,
do nothing and "exit" to skip it, or "exit 1" to abort the process.
(From OE-Core rev: c82b28982c4f630c130c827a7da3ac0454cd93b6)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make the dirty repo check somewhat less strict by ignoring old
patch directories created by this tool.
(From OE-Core rev: fea477ac55e2555c5bb0aad36db641aaa27aa915)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Add a -n option to disable component repo pull during update
* Add a 'pull' action to pull the component repos only
(From OE-Core rev: 61983b2191253b24117b63f586d5aac00c7eb48e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
&> does not work with dash - use > xxxx 2>&1 instead.
(From OE-Core rev: 5582cc58e42402c4680877e669d8c1ce058e7098)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A few recent commits in the OE-Core repository contain diffs in their
commit messages, which totally confuses git-am when applying them to the
combo repository during update. Add some code to detect and indent any
diff text in the commit message so that this does not happen (and show a
warning).
(From OE-Core rev: 6e70c95dc69be6708c3bc231cc2a99eac1360815)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If one or more components are specified for update, only check if their
repository/repositories are dirty rather than checking all of the
configured repositories.
(From OE-Core rev: b03e710c88ad0c66cf731647f26e8441d0074cae)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you specify one or more components to update immediately following
the "update" command, only these components will be updated as opposed
to the default behaviour of updating all of them.
(From OE-Core rev: 1779afce024b31edcf846329da705ed5aa9fe63e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we are running an update and the last revision hasn't changed since
the last update, don't write to the configuration file. This avoids
committing the config file with no changes other than spontaneous
reordering of sections, which sometimes occurs due to the behaviour of
the internal dictionary in Python's ConfigParser class. (This can be
fixed properly but the fix is only easy in Python 2.7+ due to the
availability there of the collections.OrderedDict class, and we
currently want to be compatible with 2.6.x as well.)
(From OE-Core rev: 4592f238de3fe6b0384c334774be9fcfc0985e4f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Avoid displaying the error from the "git status" command we use to check
the status of the config file if the config file is outside of the
repository (a situation that is already handled).
(From OE-Core rev: 9d3e54057673348e85ba479705c4ffaad56973d6)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a commit is empty (for example, commits brought over from svn where
only properties were changed) then attempting to apply it with "git am"
will result in the error "Patch format detection failed", so skip it
instead.
(From OE-Core rev: 5754bc7bcc2f57b6b56e67e2900eeaab5bc184d8)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix some grammar.
(From OE-Core rev: cc34f1bf136409d2e31c443dd5334c3a35d208af)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the config file is tracked within the combo-layer repository and it
is updated at the end of the "update" operation (because last_revision
has been changed), then automatically commit the file. This ensures that
multiple people can perform updates on different machines without the
last revision information going missing.
(If the file is outside the repository or is masked via .gitignore, this
will do nothing.)
(From OE-Core rev: 14f89acb43b0d9bec09b94668fbb056c7b5b1456)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix the dirty repository check to detect any changes, including
untracked files and uncommitted changes in the index.
(From OE-Core rev: f03fa8bbafa82ce6b09c08bf86011b6c51eb3531)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add an optional per-component branch setting to allow specifying the
branch instead of always using master.
(From OE-Core rev: 8e2b8b05607103acd539808c5ab0cc80c0d481fc)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move all example configuration to the example config file, tidy up a few
long lines and fix some grammar.
(From OE-Core rev: ec099a32243ebc7eecd86e4bf40ed38da4af3fe5)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The "init" subcommand will now copy the current state of each component
repository into the combo layer repository, set last_revision for each
component, and then use "git add" to set up the initial commit (but will
not actually make the initial commit - that is left up to the user).
Also take the opportunity to bump the version number.
(From OE-Core rev: d4d491aa118b6cbe895167c62a46d883abc7b4ab)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The combo-layer tool had trouble handling merge commits as they were
included in the revision list but not the patches; these are now
excluded from the revision list. Note however that this will not handle
merge commits that resolved a conflict; since "git format-patch" cannot
construct a linear change history over such merges there is nothing we
can currently do with these.
(From OE-Core rev: b6b6b215675631d0e505ffc8b4890f85fb3169f9)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This small patch introduces a a very simple and basic way to script
the combo-layer conf file. With that a combo can be shared with no
need to change its config - associated to the use of environment
variables for example.
*Similar* to bitbake it considers every value starting with @ to be
a python script. So local_repo could be easily configured as:
[bitbake]
local_repo = @os.getenv("LOCAL_REPO_DIR") + "/bitbake"
or any more sophisticated python syntax.
This version updates the config file description so users can be
aware of.
(From OE-Core rev: 62269642ce0e0e56d68d495b6c4d27327c9ed649)
Signed-off-by: Leandro Dorileo <ldorileo@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use --keep-cr option to "git am" or otherwise we lose carriage returns
which can be important for patches against files that use CRs.
(From OE-Core rev: bab4952075245563787293428e031fa11d6cb2b4)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds the script "combo-layer" to manipulate combo layer
repos. A combo layer repo is a repo containing multiple component
repos, e.g. oe-core, bitbake, BSP repos. The combo layer repo needs to
be updated by syncing with the component repo upstream. This script
is written to assist the combo layer handling.
The combo layer tool provides three functionalities:
- init: when the combo layer repo and component repo does not exist,
init will "git init" the combo layer repo, and also "git clone" the
component repos
- update: combo layer tool will pull the latest commit from component
repo upstream, and apply the commits since last update commit to the
combo repo. If the user specifies interactive mode(--interactive),
they can edit the patch list to select which commits to apply.
- splitpatch: split the combo repo commit into separate patches per
component repo, to facilitate upstream submission.
Combo layer tool uses a config file to define the component repo info.
Please check the combo-layer.conf.example for a detailed explanation
of the config file fields.
(From OE-Core rev: 68394476748386e58f40173643967f5a248173b1)
Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Patrick Ohly
Markus Lehtonen
João Henrique Ferreira de Freitas
Saul Wold
Richard Purdie
Paul Eggleton
Otavio Salvador
Leandro Dorileo
Yu Ke