I can't think of a reason we'd download zero sized files however there are
reasons zero length files can accidently make it onto source mirrors.
This check allows us to ignore the broken files and switch to another
mirror rather than fail with odd checksum failures.
(Bitbake rev: 300cba2e1a720dba4b83b0c76208ea93c608c1de)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to deference tags when trying to map them to commit IDs with
ls-remote. If we don't do this, a given commit might not show up
later in a specific branch. There appears to be no good reason not
to do this.
(Bitbake rev: 8ef24f4c834298348172b96ec0b855bf09552b09)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When specifying tags, they're searched for unanchored so foo/bar could
match:
refs/heads/abc/foo/bar
refs/heads/xyz/foo/bar
refs/heads/foo/bar
This change anchors the expressions so they are based against heads
or tags (or any other base level tree that has been created).
(Bitbake rev: df2e0972cd1db7abd5ec8b7cb295fb0c42e284a4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current message can be ambiguous, improve it (and also rename a
variable to clean up the rest of the function).
(Bitbake rev: 0c1bb7c0fce7b0f334311a2893ccb00385fa8d55)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a sanity check so that if some SRCREV is set and a rev parameter is given
to the url, the revision given should match.
Any tag parameter behaves the same as rev. If both are specified, error to
tell the user we're confused rather than do something which may or may not
be what they intended.
Also add some unittests for this.
(Bitbake rev: e82a4ab48991035866da9914c8b75a9bfbc9a7fc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently INVALID and None are checked as incorrect values under different
circumstances. This code standardises those checks to be consistent. We
should phase out the use of "INVALID".
(Bitbake rev: 86ef4e65ce18b71dc69643586bd2aa8f48703171)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fetcher will try:
1) PREMIRROR
2) Upstream
3) MIRROR
If it fails to download from the Upstream, but succeeds from the MIRROR,
and ud.localpath != origud.localpath (for example, the git tarball),
then we will get the error (e.g.: xf86-video-omapfb):
ERROR: Function failed: Fetcher failure for URL: 'xxx'. Unable to fetch URL from any source.
ERROR: Logfile of failure stored in: /path/to/log.do_fetch.28024
It should not show the error and let the build go on since it succeeds.
(e.g.: xf86-video-omapfb)
[YOCTO #5686]
(Bitbake rev: c08ca1e4eeb04f78e1354780cf5a4c3855e49572)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows FETCHCMD_git to override the fetcher command as the git fetcher does.
[YOCTO #5717]
(Bitbake rev: 23ab943be3a33077d6ad8be68bba53cd1e2270b4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* currently decode_url regexp parses branch=@foo as username so it ends like this:
- ('git', '', 'foo', 'git.openembedded.org/bitbake;branch=', '', {})
+ ('git', 'git.openembedded.org', '/bitbake', '', '', {'branch': '@foo'})
* http://hg.python.org/cpython/file/2.7/Lib/urlparse.py also assumes
that there is at least one '/' as separator between netloc and path,
params, so it looks reasonable to prevent including '/' in username
(Bitbake rev: 2c82742114091cb55055328b54223686816582f2)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This code clearly uses an earlier fetcher API. Update it to match master.
(Bitbake rev: e13acb4113ce75226664c3006a9776cc885e860d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This add a Git Annex backend which reuses the Git fetcher code; it
allows managing files with git, without checking the file contents
into git, being useful when dealing with files larger than git can
currently easily handle, whether due to limitations in memory, time,
or disk space.
(Bitbake rev: a61fc4db598e9d13c966712a6a0e4783e19448be)
Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For rebased git tree, some commits doesn't exist in any branch, and such commits are
valid in tag, the change is useful for such case.
(Bitbake rev: f594cb9f5a18dd0ab2342f96ffc6dba697b35f65)
Signed-off-by: Zhenhua Luo <zhenhua.luo@freescale.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was a problem:
$ bitbake xf86-video-omapfb -cfetch && bitbake xf86-video-omapfb -ccleanall
Everything should be removed, but the
0006-omapfb-port-to-new-xserver-video-API.patch.done still exists in the
DL_DIR, this is because the clean() in the fetch2/__init__.py skips
removing the local file, so that it will skip removing the .done.
The local file (file://) isn't needed to be removed since it is not
downloaded into DL_DIR, but the .done should be removed, this patch will
remove the .done, and it doesn't remove anything else since the clean()
in local.py does nothing.
[YOCTO #5687]
(Bitbake rev: 2bc99b9dfa532430a13c39fca4e5ef3a2206b3b8)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was a problem:
$ bitbake xf86-video-omapfb -cfetch && bitbake xf86-video-omapfb -ccleanall
The git2_git.pingu.fi.xf86-video-omapfb.tar.gz has been removed from the
DL_DIR, but the git2_git.pingu.fi.xf86-video-omapfb.tar.gz.done still exists,
this is because the "open(ud.donestamp, 'w').close()" in try_mirror_url() will
create the git2_git.xxx.tar.gz.done, but no one removes it (the clean() in
fetch2/__init__.py removes the DL_DIR/git2/pkg.done)
This only happens on the git fetcher AFAIK.
[YOCTO #5688]
(Bitbake rev: fb2dc84875eb477661f421b21bc404d4805ce379)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A part of fetch2/__init__.py uses 3 spaces as the indent, I
think that they should be typos.
(Bitbake rev: abafd85e2fcf23cee872e0e9e468898101430f1f)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Because of the way we were handling this error, it was printed twice -
once via logger.error() (to avoid the log being printed) and a second
time when the exception gets wrapped in a FuncFailed at a higher level.
Call logger.error() earlier and change the text we send in the
exception to be more brief, so it more closely resembles the behaviour
when there is an invalid checksum.
(Bitbake rev: 46765369d7f76ec7f67b90430131a79eb6a66235)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were checking SRC_URI md5sum/sha256sum values against None here, so
if they were set to "" then no error was produced. Since the value is
still effectively unset in this case, this is not the right behaviour;
just check if the value doesn't evaluate to False instead.
(Bitbake rev: 040943a718795c64dc4e604abfcf08b26b7d00e6)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The basecmd is initialized in urldata_init; there's no need redoing that
work.
(Bitbake rev: f8df6f746fb2e27f029a5449cee6c891b1f36f4f)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the fetcher doesn't distinguish between names that the fetcher
needs to resolve verses branch names that the user specified.
This meant that if you specify a tag and a branch, the fetcher broke. This
separates the two so that the branch name is preserved and can be used in
appropriate places.
(Bitbake rev: e85f39fe9d1b224414b5da0780da514f75c5df92)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fetcher made the rather bold assumption that if it fetched from the upstream,
the revisions were present and correct. These checks are fast and ensure that
really is the case. The avoids accidental network accessed and missing
branch configuration problems.
(Bitbake rev: a9112a102a89049cda597dad449e922c9e957a5d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In 6a48474de9505a3700863f31839a7c53c5e18a8d the url parameter to a
number of functions was removed. However, not all calls to
latest_revision() were fixed...
(Bitbake rev: 7c94ca56b2fd85a989089f58b3dcce3172a778f2)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With a SRC_URI = " \
p4://depot/folder/...;module=localfolder/localsubfolder;changeslist=${P4CHANGELIST} \
"
the subfolders of //depot/folder/... get renamed when mapped to the
local folder structure. They lose the first 3 letters. This
patch fixes that.
Issue reported by and patch sent from katutxakurra@gmail.com
[YOCTO #5380]
(Bitbake rev: 40e06dc459d9c0b5d42d65b2d2c846196fd36b1f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is no good reason to keep passing around the url parameter when
its contained within urldata (ud). This is left around due to
legacy reasons, some functions take it, some don't and its time
to cleanup.
This is fetcher internal API, there are a tiny number of external users
of the internal API (buildhistory and distrodata) which can be fixed up
after this change.
(Bitbake rev: 6a48474de9505a3700863f31839a7c53c5e18a8d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
urldata contains the url so we might as well stop passing around
pointless function parameters. This was done for legacy reasons but
its time to clean this mess up.
This is a first step in cleanup and is a standalone patch but there is
more to be done in a second patch.
(Bitbake rev: 06590cfebbcf6565a17b80cc298e3ecdfaba4656)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If an SCM mirror is in PREMIRRORS, the tarball is downloaded and then found
by the "upstream" check and handled correctly.
If an SCM mirror is in MIRRORS, the tarball is downloaded but not used
since there is no "upstream" run after MIRRORS completes. It therefore
sits there useless and unused. This code change forces the upstream to
run after a mirror tarball is found and fixes the usage of SCM mirrors
in MIRRORS.
(Bitbake rev: a66ee0994645aa5658b2f5ea134ed17d89f8751a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using git merge-base for checking for ancestors is nice but required git 1.8.0
which is not in many distrbutions yet. We therefore revert to a more ugly
check using git branch --contains until such times as we can upgrade.
(Bitbake rev: 31467c0afe0346502fcd18bd376f23ea76a27d61)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current use of git log to check if a given revision is present can be
a little fragile.
For example if revision X was on branch A, and then later added to branch
B, the update checks would not notice this since they just check for X
being in the repository.
We also had some autobuilder corruption where an older packed-refs file
was copied over a new repository containing newer pack files. There
was no update to the refs file since the revision was present but
not accessible in any branch.
The correct fix is to check that the required revisions are present
on the specific branches. This patch does this using merge-base.
(Bitbake rev: 89abfbc1953e3711d6c90aff793ee622c22609b1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Trying to use a server with username and password authentication
within the URL of the SRC_URI variable doesn't appear to work.
This patch adds the missing parts to the hg fetcher to make this
work properly.
(Bitbake rev: dc3d6d73e44802c203b3f7247f6f212acc2f69bf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have faced a corner case situation where the 'last changed
revision' returned from svn info is wrong. It happens when the last
revision is a directory move. e.g. if we assume that the svn
repository at revA has root/x/y/z/foo/bar and it is moved to
root/a/b/c/foo/bar in revB, then svn info 'last change revision' will
return revA. As such when using AUTOREV, we are going to attempt to
retrieve root/a/b/c/foo/bar (as per SRC_URI) but at revA when it did
not exist.
So this patch changes how we retrieve the latest revision and uses
'svn log --limit 1' which gives correct result in all tested cases.
(Bitbake rev: 17d8ef0b813a05c231e3dbe6e8bc82a4a9b1d2f8)
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a file ends with .xz, it currently gets overwritten during unpack:
The decompress command for .xz files is:
'xz -dc %s > %s' % (file, efile)
and as efile == file, we end up overwriting file (the source).
Fix this by adding .xz to the list of suffixes that that need to
be removed from a file name for an extract command, leaving the
bare file name. Now, for a given file foo.xz,
file == foo.xz and efile == foo, similar to how .gz .bz2 and .Z
files are treated.
(Bitbake rev: 2cd2d0a48e12ab4358fb967eaf7a56c17993f48d)
Signed-off-by: André Draszik <andre.draszik@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BB_NO_NETWORK can be set by bitbake internally by the use of
BB_FETCH_PREMIRRORONLY so update the error message to give users a
hint about this.
[YOCTO #3222]
(Bitbake rev: cac3060d0bf8c7deeacda18d06d92787911380d0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
AUTOINC was meant to appear once at the start of the version string.
The list of names may not be sorted meaning it could get inserted in
the middle. This patch simplifies the code and ensures it appears at
the start.
Include cache version bump to ensure the cache picks up these changes.
(Bitbake rev: ad8bf10d873abb94d987860a3f6d06b134fb8a99)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The assignment to True was missing from the code, well spotted Saul!
(Bitbake rev: e493fe8cb4953935f01361ffc0240e5818ebb283)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The long srcrevs are mainly used or the workdir construction as well as
the package version. The long entries are hashes generated by the git fetcher
and other scms using a similar revision mechanism.
We need these to change when the package changes however collisions are
unlikely to happen within the domains we care about. The long revisions
have generated negative user feedback due to the use in path and file
names.
This patch therefore truncates the revisions to 10 characters maximum.
This should be safe in the contexts where these revisions are used as
the chances of spatially close collisions is very low (distant
collisions are not a major issue in the way we use these).
(Bitbake rev: 43a8319cda7fae37862dae323eeb24cb39ca21b7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now we no longer try and provide increasing values from the fetcher,
we can simplify the function structure for the sortable_revision
pieces and move the AUTOINC handling directly into the function
which needs it, simplifying the code.
(Bitbake rev: fb068bee47bb1a06f02447daf16c2b2a79c03288)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move various random imports to the start of the modules as cleanup
and avoid an import issue with bb.process on python 2.6.
(Bitbake rev: aed4adfbe3a591ca4f8e41fb763c9f961bf2e6d5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* with read-only PREMIRROR (e.g. mounted over NFS or CIFS
and referenced as file:///mnt/premirror) we cannot use
BB_GENERATE_MIRROR_TARBALLS because all git2_abc.git.tar.gz
files later became just symlinks to read-only location in PREMIRROR
(it works fine on first build and for new components, because
at that time there isn't tarball on PREMIRROR yet).
ERROR: Fetcher failure: Fetch command failed with exit code 141, output:
tar (child): /build/downloads/git2_abc.git.tar.gz: Cannot open: Read-only file system
tar (child): Error is not recoverable: exiting now
(Bitbake rev: 3627b02f77c78beedadadd77c619b9e5edaae076)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* this can be useful when someone wan't to compare old file with
bad checksum and new one
(Bitbake rev: 33c6b93597dd43ab03ce7b62ba3eeb1893a68c38)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds very basic git submodule support to the fetcher. It can be
used by replacing a git:// url prefix with a gitsm:// prefix, otherwise
behaviour is the same as the git fetcher. Whilst this code should be
functional, its not as efficient as the usual git fetcher due to the
need to checkout the tree to fetch/update the submodule information. git
doesn't support submodule operations on the bare clones the standard git
fetcher uses which is also problematic.
This code does however give a starting point to people wanting to use
submodules.
(Bitbake rev: 25e0b0bc50114f1fbf955de23cc0c96f5f7a41e3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bb.process.run() will return one tuple, e.g:
p4file = ('strA\nStrB\nstrC\n'), then there will be an iteration on p4file:
for i in p4file:
[snip]
The i will be 's t r A ...', this is incorrect. use splitlines() to fix
the problem.
[YOCTO #3619]
(Bitbake rev: b7440fb36b419996046f607e66434ce34722272b)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit 21fe2683aefde10e847e66c11c26d4f4c1e07cfd
since bitbake-selftest doesn't pass when this is applied and
we're seeing multiple build failures from this change.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie
Robert Yang
Martin Jansa
Zhenhua Luo
Otavio Salvador
Paul Eggleton
Olof Johansson
Peter Kjellerstedt
Florin Sarbu
Volker Vogelhuber
Nicolas Dechesne
André Draszik
Vanveerdeghem, Simon
Matthew McClintock