When specifying tags, they're searched for unanchored so foo/bar could
match:
refs/heads/abc/foo/bar
refs/heads/xyz/foo/bar
refs/heads/foo/bar
This change anchors the expressions so they are based against heads
or tags (or any other base level tree that has been created).
(Bitbake master rev: df2e0972cd1db7abd5ec8b7cb295fb0c42e284a4)
(Bitbake rev: da93afe9834e137ed1e9410380181286c80198b5)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* with read-only PREMIRROR (e.g. mounted over NFS or CIFS
and referenced as file:///mnt/premirror) we cannot use
BB_GENERATE_MIRROR_TARBALLS because all git2_abc.git.tar.gz
files later became just symlinks to read-only location in PREMIRROR
(it works fine on first build and for new components, because
at that time there isn't tarball on PREMIRROR yet).
ERROR: Fetcher failure: Fetch command failed with exit code 141, output:
tar (child): /build/downloads/git2_abc.git.tar.gz: Cannot open: Read-only file system
tar (child): Error is not recoverable: exiting now
(Bitbake rev: 3627b02f77c78beedadadd77c619b9e5edaae076)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds very basic git submodule support to the fetcher. It can be
used by replacing a git:// url prefix with a gitsm:// prefix, otherwise
behaviour is the same as the git fetcher. Whilst this code should be
functional, its not as efficient as the usual git fetcher due to the
need to checkout the tree to fetch/update the submodule information. git
doesn't support submodule operations on the bare clones the standard git
fetcher uses which is also problematic.
This code does however give a starting point to people wanting to use
submodules.
(Bitbake rev: 25e0b0bc50114f1fbf955de23cc0c96f5f7a41e3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- do not use the BB_URI_LOCALCOUNT database for computing revision
incremental numbers anymore
- sortable_revision now generates "AUTOINC+${latest_rev}"
- use one incrementing value rather than several
- PV becomes 0.1+gitAUTOINC+deadbeefdecafbad_decafbaddeadbeef
- remove all localcount code and simplify the fetcher
- this patch addresses the following proposal:
http://lists.linuxtogo.org/pipermail/bitbake-devel/2012-November/003878.html
(Bitbake rev: 61cf01c5c236b4218f40cfae7c059c2b86765dbd)
Signed-off-by: Constantin Musca <constantinx.musca@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Doc cleanup, no functional change.
(Bitbake rev: 5161a84f5dcfe748382a5073349bf10ed21641f9)
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake-selftest is failing due to directories not being created. This adds in an
appropriate mkdir so the tests can complete. Presumably in general OE use, something
else is ensuring the parent directory is created.
(Bitbake rev: 1270a07713e2a6c6e6fadcc61b785aebc99ae17b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you have foo and foo.git in GITDIR, the two can end up being confused
by git with some horrible union of the two being cloned. This adds
a workaround to avoid this happening until git 1.7.9.2 onwards is
common enough for this to be removed. We use a symlink to hide
the directories we don't want git to know about.
(Bitbake rev: bbf1f6fe594c721a296ca09ee7c583d4a205c591)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
'git remote prune' at this location does not make much sense because
the following 'git remote rm' will prune stale and non-stale branches.
The 'prune' can cause trouble because it will access the network
bypassing the no-network code in bitbake. When this operation fails and
throws an exception, the next command (--> 'git remote rm') will be
skipped. This in turn, will make all the following operations fail,
because they assume that the remote does not exist yet.
(Bitbake rev: 2ba23df5fad4b94d38a6aed97f7822226d72eb89)
Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If '*' does end up in mirror urls accidently, some strange things
can break since supports_checksum() looks for this, ud.localpath can
then get ignored and this can lead to empty directories being downloaded
"successfully". '*' is a special case for file urls only at this point
so remove any entries that accidentlly make it in through url mapping.
(Bitbake rev: 1369bec2404d942acc3618a8d005ec6868dcfd41)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If -l is specified and the source and destination are not on a common
filesystem an error occurs. The -l option is however the default for
git for local paths which the fetcher already now ensures in the
file:// case.
We can therefore safely drop the -l option.
(Bitbake rev: 3aeb268b2aaab4bb8b1cfff1450e0b76aa8ce855)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A file:// url should use "clone -l" to greatly speed
up the clone in the case of a kernel when it is local.
(Bitbake rev: 2bab2cc3ffe67ee2a308074a6e4c2c7be5636d2f)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There were some hardcoded behaviours in the system for which backends
support checksums verses which backends recommend them verses which
don't recommend them.
This moves the functionality into specific fetchers and then makes the
general code generic. This cleans up the codebase and fixes some corner
cases such as trying to checksum directories returned by the git fetcher.
(Bitbake rev: ef6d268f7b8527541a7fb044cf95a973be4097f4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With floating revisions and no specified branch, the fetcher could fail
with some obtuse errors. This was due to the branch name being set to None
which makes no sense. This patch reworks some conditions to avoid this.
(Bitbake rev: 740c58d43cfb1445dd126e4827bb70ce988ca107)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Don't check for network access before grabbing the the current head,
cloning, or updating a clone when the protocol is 'file'.
(Bitbake rev: d5847bc5254b9d2f28a6b574f6157d1286add27c)
Signed-off-by: Jeff Polk <jeff.polk@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For similar reasons as the nocheckout option, packages that need
enhanced control over the checkout and branch creation on a repository
may want a complete mirror/bareclone created of the repository when
performing the unpack.
This is useful/required when a local respository is being used, but
local tracking branches have not been created for all branches that
a given recipe needs to manipulate. The standard git clone operations
will create remote branches for the branches that are local to the
source repository, but branches that are remote do not translate to
the destination repository. Doing a mirror/bare clone of the source,
makes all branches available to the repository.
This is a particular use case, but the ability to do a bare clone
creates great flexibility in recipe space, with no impact to recipes
that don't need this functionality.
To implement this, a new option 'bareclone' is craeted which creates
a mirror copy of the repository and leaves it bare in the unpacking
phase. A recipe that uses this option must both checkout and debare
the repository itself.
(Bitbake rev: 82482aae6f311c994275fb0b6b32d954bbfc78c3)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To quote my report of this to the git mailing list:
"""
I have a problem with git clone commands using alternates failing by
mixing up different repositories. I have a situation where I could end
up with both:
/srv/mirrors/repo
/srv/mirrors/repo.git
as bare clones.
I then try cloning "repo" with alternates with the command:
$ git clone -s -n /srv/mirrors/repo /tmp/foo
Cloning into /tmp/foo...
done.
$ cat /tmp/foo/.git/objects/info/alternates
/srv/mirrors/repo.git/objects
Note how I'm now referencing repo.git, not repo. This doesn't work as
expected giving some very bizarre results when actually using the
repository.
I appreciate this is a rather bizarre corner case but its one that is
breaking the build system I work with. Ideally people would use a
consistent URL for the same repository but we have an example where they
haven't and this really shouldn't break like this.
Looking at the code, the cause seems to be
clone.c:get_repo_path():
static char *suffix[] = { "/.git", ".git", "" };
since its looking in order for:
repo/.git (fails)
repo.git (suceeds, incorrect)
repo (never looked at)
I'm not sure what would break if that order were to change, swapping the
last two options.
I can "force" the issue by running:
git clone -s -n /srv/mirrors/repo/ /tmp/foo
but this results in the slightly odd looking:
$ cat /tmp/foo/.git/objects/info/alternates
/srv/mirrors/repo//objects
which does at least work.
"""
This patch adds the trailing slash to ensure the correct repository is
referenced at the expense of some ugliness in the alternates file.
(Bitbake rev: d978e7b35550e3785c7c567ffe4c40a3c3947450)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Local cloning of git repositories from DL_DIR into WORKDIR fails when
using ssh URL with port specification e.g.
"ssh://user@host:port/path/to/repo.git". Git fetcher clones such remote
repository into "${DL_DIR}/git2/host:port.path.to.repo.git". However,
when clonging from ${DL_DIR}/git2/host:port.path.to.repo.git into
${WORKDIR}, git fetcher fails with "ssh: Could not resolve hostname
${DLDIR}/git2/host: Name or service not known".
A solution is to replace ":" by "." in host component, similarly as it
is done when replacing "/" with "." in path component, so that local
clone directory in DL_DIR looks like this: "host.port.path.to.repo.git".
(Bitbake rev: 1f2867b79f1cd2bfbdc849ba5677a39db6fa3396)
Signed-off-by: Juraj Hercek <juraj.hercek@jhksoftware.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
git fetches can fail (or at least return failed) when trying to
fetch and prune rebased branches. This patch simply adds a -f
to the git fetch command so these failure are ignore
Generally, if some SHA was rebased away it's not coming back so
there is no point in not doing this force
(Bitbake rev: a7b75e4db52445b30ec0fc0053dcf454f5f7d2db)
Signed-off-by: Matthew McClintock <msm@freescale.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make the git fetcher's subpath (path within the git repo to fetch)
option set the destsuffix (destination directory) option by default.
This reverts the behaviour of subpath to the same as when it was
introduced.
Based on a patch by Dmitry Eremin-Solenikov <dbaryshkov@gmail.com>
(Bitbake rev: 3e7f8afeacf7c8c8de3e87778a3907e33d4a06b3)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* in some cases there could be output like this
ERROR: ld.so: object 'libpseudo.so' from LD_PRELOAD cannot be preloaded: ignored.
before wc -l output and returned 'output.split()[0] != 0' is always True
(Bitbake rev: 3a54dcc09a12406ec6cf22b4b1a2cc4fc203822c)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* logging/logger typo was fixed in 38a598731b49c8a0ba0ede570adc33eb1e848235
but debug level is still missing
(Bitbake rev: 9de432fe2348cdbc93037bb49abb508d1fd38336)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 639db8c766cada7180f9447f51303f9b30d7e817)
Signed-off-by: Holger Hans Peter Freyther <holger@moiji-mobile.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a user has a ~/.gitconfig file, git fetch --all will reference it. To avoid
this we should run git fetch with an explicit url telling it to fetch all
references (which includes tags).
I'm assured this means git won't reference the file, see the discussion on the
git mailing list (subject Overriding ~/.gitconfig using GIT_CONFIG).
[YOCTO #1134]
(Bitbake rev: 8540c09d4e509e3277940464c71174d8a0aca6ab)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use git ls-remote to implement checkstatus command for the git fetcher.
(Bitbake rev: 0ed281feb6d244d3700da484f4e83394ae394f93)
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the git fetcher can malfunction when branches change in remote
repositories since whilst the update code updates the "origin" remote,
this isn't linked to the local heads.
By passing the --mirror option to 'git clone' and 'git remote add',
linkage between the local heads and remote heads is created with a 1:1
mapping, hence all the appropriate heads are then updated correctly.
This fixes some issues which have been seen with the Yocto autobuilder
mirrors.
(Bitbake rev: 3725602ec53df116dc108b3197a426b86ca43d5f)
Signed-off-by: Richard Purdie <richard.purdie@linux-foundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When an invalid 'protocol' parameter is used in a git SRC_URI,
the error reported was not helpful:
ERROR: Function 'Fetcher failure for URL: 'None'.
<environment dump>
fatal: Could not make temporary directory: No such file or directory
So instead check that ud.proto is set to something valid, and if not
raise a meaningful ParameterError which explains that the protocol
type is the source of the problem.
This fixes bug [YOCTO #1142]
(Bitbake rev: a2a29b72275ab03a263f4479a590b92111a0d6a8)
Signed-off-by: Scott Garman <scott.a.garman@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The git command string logged via check_network_access() does not match
the actual command executed in a few places. Ensure that it does.
(Bitbake rev: 10f3ca52dc274cd8b240987cfd7cd003aeda7ab1)
Signed-off-by: Darren Hart <dvhart@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Replace a call to print() with logging.debug() and flesh out the
message to clarify the state being reported.
(Bitbake rev: 9a28f7744e2f4224e7c097b8c4c1d49731b9a47e)
Signed-off-by: Darren Hart <dvhart@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
current git fetcher unpack method only checkout index and working tree,
but did not did not update the git branch in ref/heads, so user may not
get right info in ${S} by using git.
this patch enhance the unpack by using git checkout to fix this issue.
Fix bug [YOCTO #1089]
(Bitbake rev: c0eb89054aef4957966f98b44e7f3cce14fb337a)
Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
make the nocheckout option format to be: default is "0",
use nocheckou=1 to set this option
with this patch, the format will be consistant with other bitbake options
like rebaseable
(Bitbake rev: bd51659f5ee521cb8e6631d5f26792ab573e6b30)
Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some upstream git repo may rebase in the future, which means current
revision may disappear from the upstream repo after the rebase.
current git fetcher can not handle this case, because the git mirror
tar ball is per repo, and may also change in the rebase and lost the
current revision info.
To fix this issue, this patch
- add rebaseable tag in the SRC_URI
- for rebaseable repo, make git mirror tar ball per revision, in this
case, even upstream rebase, the git mirror still has the current
revision info.
- for rebaseable repo, generate mirror tar ball by default, since the
repo may change in the future.
(Bitbake rev: 92701d4c5372db48847c70da4ebd0736d79fd54b)
Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In current git fetcher, tag does not work due to commit http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=5920e85c561624e657c126df58f5c378a8950bbc. Tag is not in sha256 form, so it will be treated invalid, and silently replaced by latest revision.
To fix it, this patch treat tag name as branches name, thus it will be handled correctly later. Thanks Richard for reviewing and proposing the better approach.
Fix [YOCTO #972]
CC: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Yu Ke <ke.yu@intel.com>
The ordering constrains on the urldata_init functions are not straight
forward. To avoid further problems, create a helper function to setup
the source revisions which the init functions can all at the appropriate
point.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix a bug where ud.branches were being referenced before it was set by
the git fetcher when using AUTOREV. To do this some ordering needed
to be changed. This fixes errors like:
ERROR: Error parsing /recipes-kernel/linux/rt-tests_git.bb: Failure expanding variable
SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception
AttributeError: 'FetchData' object has no attribute 'branches'
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* This patch fixes a cosmetic issue currently we get with master
WARNING: /home/kraj/work/bitbake/lib/bb/fetch2/__init__.py:733:
DeprecationWarning: Call to deprecated function bb.mkdirhier: Please use bb.utils.mkdirhier instead. bb.mkdirhier("%s/%s" % (rootdir, destdir))
(Bitbake rev: 36fe59ce314c295d239b76de34c8714def2c32d5)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This create a clean() method in each of the fetcher modules
and correctly cleans the .done stamp file and lock files
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Richard Purdie
Martin Jansa
Constantin Musca
Robert P. J. Day
Enrico Scholz
Jason Wessel
Jeff Polk
Bruce Ashfield
Juraj Hercek
Matthew McClintock
Paul Eggleton
Holger Hans Peter Freyther
Joshua Lock
Scott Garman
Darren Hart
Yu Ke
Khem Raj
Saul Wold