Starting from tar 1.29 the --exclude option won't work
anymore if is not used before the path. There are some
fetch modules that copy the ptest using tar and --exclude
option. This fixes these for bitbake.
[YOCTO #9763]
(Bitbake rev: cc71d5d9da71ea5f21d02f3b2fbf119bd2d794f0)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
While switching from master to krogoth build with a common download directory,
got a large number of warnings like the one listed below:
WARNING: freetype-2.6.3-r0 do_fetch: Couldn't load checksums from
donestamp /home/maxin/downloads/freetype-2.6.3.tar.bz2.done: ValueError
(msg: unsupported pickle protocol: 4)
These warnings are caused by the difference in pickle module
implementation in python3(master) and python2(krogoth). Python2 supports
3 different protocols (0, 1, 2) and pickle.HIGHEST_PROTOCOL is 2 where as
Python3 supports 5 different protocols (0, 1, 2, 3, 4) and
pickle.HIGHEST_PROTOCOL is obviously 4.
My suggestion is to use 2 since it is backward compatible with python2
(all the supported distros for krogoth provides python2 which supports
pickle protocol version 2)
(Bitbake rev: cc67800f279fb211ee3bb4ea7009fdbb82973b02)
Signed-off-by: Maxin B. John <maxin.john@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using a PREMIRROR with plain (non-unpack) files, a SRC_URI like
SRC_URI = "file://devmem2.c"
will cause devmem2.c to be a symlink in the WORKDIR pointing to the
local PREMIRROR.
Trying to apply a patch on this file will either modify the file on
the PREMIRROR or will fail due to sanity checks:
ERROR: devmem2-1.0-r7 do_patch: Command Error: 'quilt --quiltrc /cache/build-ubuntu/sysroots/x86_64-oe-linux/etc/quiltrc push' exited with 1 Output:
Applying patch devmem2-fixups-2.patch
File devmem2.c is not a regular file -- refusing to patch
(Bitbake rev: cfd481fe9799e7a4c6bfac32e56cc91cfcd81088)
Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Implement progress reporting support specifically for the fetchers. For
fetch tasks we don't necessarily know which fetcher will be used (we
might initially be fetching a git:// URI, but if we instead download a
mirror tarball we may fetch that over http using wget). These programs
also have different abilities as far as reporting progress goes (e.g.
wget gives us percentage complete and rate, git gives this some of the
time depending on what stage it's at). Additionally we filter out the
progress output before it makes it to the logs, in order to prevent the
logs filling up with junk.
At the moment this is only implemented for the wget and git fetchers
since they are the most commonly used (and svn doesn't seem to support
any kind of progress output, at least not without doing a relatively
expensive remote file listing first).
Line changes such as the ones you get in git's output as it progresses
don't make it to the log files, you only get the final state of the line
so the logs aren't filled with progress information that's useless after
the fact.
Part of the implementation for [YOCTO #5383].
(Bitbake rev: 4027649f422ee64b1c4e1ad8d48ac295050afbff)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some services such as SourceForge seem to struggle to keep up under load, with
the result that over half of the autobuilder checkuri runs fail with
sourceforge.net "connection timed out".
Attempt to mitigate this by re-attempting once the network operation on failure.
(Bitbake rev: 54b1961551511948e0cbd2ac39f19b39b9cee568)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In recipes which use the perforce fetcher, enable use of SRCREV to
specify any of: ${AUTOREV}, changelist number, p4date, or label. This
is more in-line with how the other fetchers work for source control
systems.
Allow p4 to use the P4CONFIG env variable to define the server URL,
username, and password if not provided in a recipe.
This does change existing perforce fetcher usage by recipes and will
likely need those recipes which use the perforce fetcher to be updated.
No recipes in oe-core use the perforce fetcher.
References [YOCTO #6303]
(Bitbake rev: 6298696bb94a127cdec7964315f6891ba92cd026)
Signed-off-by: Andrew Bradford <andrew.bradford@kodakalaris.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Python 3 changed the return value of check_output to binary rather than
a string. This fix decodes the binary before calling splitlines, which
requires a string.
(Bitbake rev: 1072beefe172423873a22a10c7171e10d0401e1e)
Signed-off-by: Stephano Cetola <stephano.cetola@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
deb packages in modern Debian versions have the data tarball compressed
with xz rather than gzip, and thus explicitly extracting data.tar.gz
fails. Unfortunately ar doesn't support wildcards matching items to
extract, so we have to find out what the name of the file is first and
then extract it, relying on tar to figure out how to unpack it based on
the filename rather than doing it with pipes and making that
determination ourselves.
(Bitbake rev: 17ff08d225a8fa7faffd683c028369574954fba9)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously the code used to match a reference to its SHA-1 in
_latest_revision() used the Python "in" operator, which made it match
if the reference matched the beginning of an existing tag or
branch. This test, however, must be exact. I.e., either the reference
matches a tag or branch exactly, or it does not match at all.
(Bitbake rev: e5986c78a6108fd7578989c20efcbf0b81c97e03)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Various misc changes to convert bitbake to python3 which don't warrant
separation into separate commits.
(Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this, the dict can reorder causing sanity test failures.
(Bitbake rev: ca8c91acc9396385834b266d4e8b84d917e5e298)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
python deprecated logger.warn() in favour of logger.warning(). This is only
used in bitbake code so we may as well just translate everything to avoid
warnings under python 3. Its safe for python 2.7.
(Bitbake rev: 676a5f592e8507e81b8f748d58acfea7572f8796)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BB_ORIGENV value on the datastore can be NoneType thus raising an AttributeError
exception when calling the getVar method. To avoid this, a check is done before
accesing it.
[YOCTO #9567]
(Bitbake rev: f368f5ae64a1681873f3d81f3cb8fb38650367b0)
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some users may want to use authenticated SSH connections with credentials stored
in a keyring, such as gnome-keyring. These typically need a DBus session bus
connection, so pass DBUS_SESSION_BUS_ADDRESS into the fetcher environment.
To avoid the user needing to set it in their local.conf (which wouldn't be
usable) or adding it to the environment-cleansing whitelist (which would
potentially impact builds) allow the variables being passed to the fetchers to
come from the data store (first) or the original environment (second).
(Bitbake rev: 20ad1ea87712d042bd5d89ce1957793f7ff71da0)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It was used for workaround git 1.7.9.2 which was released in 2012 which
should not be existed on nowadays host, so remove it to avoid
confusions.
(Bitbake rev: 6140d0cc9aecf1029ca16fed47071dfcc92f4269)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake would fail to classify the following URL as belonging to a
allowed network, because of the port number in the url.
BB_ALLOWED_NETWORKS = "*.example.com"
SRC_URI = "http://git.example.com:8080/foo.tar.gz"
Since protocols aren't specified in the BB_ALLOWED_NETWORKS variable,
it's reasonable to believe that this should work regardless of protocol
being used.
(Bitbake rev: ff603df23037e10fb2cfdf150429cba3f65072cd)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are some codepaths where the file checksum is verified and can
be found to mismatch but the 'rename' logic doesn't kick in. If code
relies on the presence of a file for the checksum having been checked
(e.g. uninative.bbclass) then it can be used when the checksum hasn't
matched.
Therefore rename the file whenever an invalid checksum is encountered.
(Bitbake rev: 69ef6c8a9db02bfa0e3fac72481ec26586a29a01)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When prefix is part of the version directory it need to ensure that
only version directory is used so remove previous directories if exists.
Example: pfx = '/dir1/dir2/v' and version = '2.5' the expected result
is 'v2.5' instead of '/dir1/dir2/v2.5'.
[YOCTO #8778]
(Bitbake rev: c760531c6dbf88135ab9f8e6f0784ccbf2cce1e4)
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Little improvement for reference tokens by names instead of index.
(Bitbake rev: e8ea15eeb1857ed4bb6337836bd2fb1f5dbb1bdf)
Signed-off-by: Aníbal Limón <limon.anibal@gmail.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We recently dropped lockfiles for file:// urls which in itself makes
sense.
If a file url redirects to something like an http:// mirror, we'd have
no lock taken for the original file and could race against others
trying to download the file. We therefore need to ensure there is a
lock taken in the mirror handling code.
This adds code to take such a lock, assuming it isn't the same lock
as the parent url.
(Bitbake rev: 913b6ce22cd50eac96e8937c5ffc704bfce2c023)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fixes a problem when the repository contains multiple levels of submodules via a resursive submodule init.
(Bitbake rev: dbafbe229360ffe5908b106a9c10e274712b9b17)
Signed-off-by: Derek Straka <derek@asterius.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This makes it possible to prevent a recipe to be cached, and thus,
parsed every time.
Use with care.
[YOCTO #8853]
(Bitbake rev: 78335c1fbe5266116700c2413aac28b00423a75b)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes [YOCTO #9231]
npm when given an invalid registry URL with --registry actually goes and
fetches from the default registry, but this commit makes sure it goes to the
specified one.
(Bitbake rev: 7c849be7c70a5db4f66fe3041486abb923b5e4ee)
Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For some reason, the values for SRC_URI[md5sum] and SRC_URI[sha256sum]
were not being expanded. That lead to the following code not working
as expected:
SRC_URI = "http://.../${PN}-${PV}.tar.gz"
MD5SUM = "123abc..."
SHA256SUM = "abcd1234..."
SRC_URI[md5sum] = "${MD5SUM}"
SRC_URI[sha256sum] = "${SHA256SUM}"
(Bitbake rev: ba011470df0ea8bd89f01c0b02ec4b3969e60ce7)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this you get a rather odd traceback instead of the proper
exception message.
(Bitbake rev: 2fe1826d3077eeda6cde433d3a1e6620f74e08dd)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The output of "npm view dependencies" isn't entirely JSON if there are
multiple results, but the code here was just discarding the output if
the entire thing didn't parse as JSON. Split the output into lines and
iterate over it, parsing JSON fragments as we find them; this way we end
up with the last package's dependencies since it'll be last in the
output.
Digging further, it seems that the dependencies field reported by "npm
view" also includes optional dependencies. That wouldn't be a problem
except some of these optional dependencies may be OS-specific; for
example the "chokidar" module has "fsevents" in its optional
dependencies, but fsevents only works on MacOS X (and is only needed
there). If we erroneously pull in fsevents, not only is it unnecessary
but it causes "npm shrinkwrap" to throw a tantrum. In the absence of a
better approach, look at the os field and discard the module (along with
any of its dependencies) if it isn't for Linux.
As part of this, we can reduce the calls to npm view to one per package
since we get the entire json output rather than querying twice for two
separate fields. Overall the time taken has probably increased since we
are being more thorough about dependencies, but it's not quite as bad as
it could have been.
(Bitbake rev: 436d67fe7af89ecfbd11749a6ae1bc20e81f2cc8)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
"2 || 3" is a valid version specification for a dependency in an npm
package.json file, but of course that looks like something else when
sent to a shell. Quote the version value to avoid this.
(Bitbake rev: bea0246831a46d943d2e27d6b38f6e498bd3413c)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tarballs that are fetched down via npm repositories seem to often have
unknown headers. This doesn't affect our ability to extract the contents
though so we don't really care to see those warnings.
(Bitbake rev: b38975103e52a0c25e9ad9032c8cca1c47cbdcc2)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
npm allows you to specify a dependency as a Github username+path, or
omit the version entirely. You can hit these if you don't use a
shrinkwrap file, with the result that the code later fails due to the
output of "npm view" being empty; so handle this lazily by just ignoring
this part of the dependency if it's not really a version.
(Bitbake rev: 7b7a65c44dbdd5ba9366d4e2093f76df8758d546)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
No code changes, just fix to use four spaces.
(Bitbake rev: 66a9ee7d54ca9c25209f72da079f260ccdcc872a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
npm-shrinkwrap will sometimes resolve a git URL which instead of a http url, in
this case go and grab the dist.tarball via npm instead of using the resolved
URL.
(Bitbake rev: eb53b927ff59aa19cf28bc46beb9f9a185a59990)
Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous commit breaks absolute pathnames in file:// urls, this
fixes it.
(Bitbake rev: b8113a1800687a37a26ac28deafdbafd74cc138e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When in SRC_URI appears file://dir;subdir=foo unpacker copies 'dir' to ${WORKDIR}, not
${WORKDIR}/foo as it should be.
These changes are fixing following bugs as well:
Bug 6128 - Incorrect wildcard unpack behaviour in fetcher
Bug 6129 - Local directories unpack to a different location than local files
(Bitbake rev: e659a3b0c2771679057ee3e13cd42e6c62383ff2)
Signed-off-by: Alexander Shashkevich <alex@stunpix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For some reason the enablement piece of the patch went missing, add it.
(Bitbake rev: 0270b5a3873ed0aeca3a66198c87a6164fb644b8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For local files, there are no races with downloads, we don't need ".done"
stamps and we don't need lockfiles.
This considerably cleans up DL_DIR and all the pointless ".done" files
as well as removes stalls over local files with the same name.
(Bitbake rev: 48e903745db578d9b9b425a8d411c1369df0eb94)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than create ".lock" and ".done" files with no name, error,
forcing us to fix the cases where this is a problem.
(Bitbake rev: 81158071508cc68c39db7d501370872f44d335cc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we specify urls such as npm://somehost;someparams the fetcher currently
does a poor job of handling mirrors of these urls due to deficiencies in the
way decodeurl works. This is because "somehost" is returned as a path, not
a host.
This tweaks the code so that unless its a file url, the host is returned
correctly.
This patch also adds test cases for these urls to the exist set of test
urls.
We need to tweak the URI() class since this thinks this is a relative url
which is clearly isn't. We also need to handle the fact that encodeurl will
error if passed a url of this form (it would want the path to be '/'.
(Bitbake rev: 83203cd2e677706e0111892a7843b83263cb8bd9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we wget something which looks directory like we end up with lock files
and done stamps without names, they also all use the same lockfile.
This change ensures that we use separate lock files based on the url
and avoid creating the mysterious ${DL_DIR}/.done files.
(Bitbake rev: 20bc82086018832e047345a672d74b6c1c113650)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
npm fetcher with support for shrinkwrap files and lockdown files to easily
download and install an npm package with strict dependency resolution.
The SRC_URI should be in the format of:
SRC_URI = "npm://registry.npmjs.org/;name=${PN};version=${PV}"
To add a shrinkwrap and lockdown file use:
NPM_SHRINKWRAP := "${THISDIR}/${PN}/npm-shrinkwrap.json"
NPM_LOCKDOWN := "${THISDIR}/${PN}/lockdown.json"
(Bitbake rev: dec75bbc5d075acb322dad8b1c40d6bd518dc9fd)
Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move the local file checksum functionality from bb.fetch2 into
bb.checksum module.
(Bitbake rev: 4f60933283f377d68f191db849dac6c1dc7a0aed)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Drop unused 'd' argument from the cache save methods, simplifying the
API.
(Bitbake rev: 81bc1f20662c39ee8db1da45b1e8c7eb64abacf3)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to use in other modules since is a common function
when needs to get proxies working.
(Bitbake rev: 85c529044381895556d603a3974de22392646a22)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since people do copy and paste these things, clean up old syntax styles.
(Bitbake rev: 4fb028b0bd14d3e4b3fd7a89c643528728566476)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This shouldn't be in here, use a variable instead.
(Bitbake rev: 2e25d09a1ab62ccc3573d13114d59838cf4b07f2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This sets the scene for removing the default False for expansion from
getVarFlag. This would later allow True to become the expand default.
On the most part this is an automatic translation with:
sed -e 's:\(\.getVarFlag([^,()]*, [^,()]*\)):\1, False):g' -i `grep -ril getVar *`
There should be no functional change from this patch.
(Bitbake rev: 7c3b99c6a716095af3ffce0b15110e91fb49c913)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fix a problem when checking out a commit that changes the submodules
previously checkout.
Example:
Recipe uses branch A and then it updates to use branch B, but branch B has
different submodules dependencies then what branch A previously had.
(Bitbake rev: 54a3864246f2be0b62761f639a1d5c9407aded4f)
Signed-off-by: Felipe F. Tonello <eu@felipetonello.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The recent change to verify_checksum() to only show checksum warnings
if no checksums are supplied made it possible to simplify the logic a
bit more.
(Bitbake rev: 1dc00b874acae44bbba9d8028d94f7bc97ddcd76)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The core change here is to fall back to GET requests if HEAD is rejected in the
checkstatus() method, as you can't do a HEAD on Amazon S3 (used by Github
archives). This meant removing the monkey patch that the default method was GET
and adding a fixed redirect handler that doesn't reset to GET.
Also, change the way the opener is constructed from an if/elif cluster to a
conditionally constructed list.
(Bitbake rev: 6ec70d5d2e330b41b932b0a655b838a5f37df01e)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If one checksum is supplied to a SRC_URI, we really don't want to show
warnings about the other type which isn't present as one checksum
is really good enough for most cases.
(Bitbake rev: 43358a9b595b2928458a5f463cf1949394160c3a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit 4193e99adce8e88f12ac88d7578ad39575f7e346.
It seems the underlying issue was caused by ":" in the url which isn't
supported. The patch was therefore incorrect.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit e130dca85bac82bd4d88f94a6bf9fe36e8ad4d7c.
This is in fact a valid use case, for example the sstate.bbclass code
sets up SSTATE_MIRRORS as PREMIRRORS. Its quite common to map those
file:// urls to remote http:// urls and with the above change, this
no longer works.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This goes undetected most of the time, but when updating a repository,
if the ud.fullmirror file is not present, you end up getting an
exception instead of carrying on because the errno module is not
loaded (specifically "if exc.errno != errno.ENOENT").
(Bitbake rev: e6fca8480731ce817df9bee61438347a5e3d3017)
Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some servers, e.g. bitbucket.org can't cope with ssh:// as part of
the git url syntax. git itself is happy enough with this but you
get server side errors when using it.
This changes the git fetcher to use the more common ssh url format
which also means we need a : before the path.
Seems a shame to have to do this due to broken servers however
it should be safe enough since this other form is the one most people
use on the commandline so it should be safe enough.
[YOCTO #8864]
(Bitbake rev: 4193e99adce8e88f12ac88d7578ad39575f7e346)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
7-Zip is a file archiver claiming the highest compression ratio.
This patch allows using 7-Zip commpressed files in bitbake recipes.
Two common formats are supported:
SRC_URI = "file://abc.tar.7z"
SRC_URI = "file://abc.7z"
(Bitbake rev: 7120f5bfaae54e91bc95da5667831424724ce613)
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The PREMIRROR isn't useful for "file://", so avoid using it, this is
good for searching speed and can reduce useless lines in log.do_fetch.
(Bitbake rev: e130dca85bac82bd4d88f94a6bf9fe36e8ad4d7c)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When a mirror tarball is fetched, the original fetch method is called, which
unpacks the mirror tarball. After the original method is called, it checks the
localpath of the mirror tarball rather than the clone path, which isn't ideal,
particularly if the mirror tarball was removed due to being out of date. We
know the original fetch method will do what it needs to do to get its content
in the form it needs from the mirror tarball, so we can use its localpath
instead.
(Bitbake rev: 1732ad65d6c7d67b7d07cb30c074f5016adadbea)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Awais Belal <awais_belal@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rename REGEX to UPSTREAM_CHECK_REGEX, REGEX_URI to UPSTREAM_CHECK_URI, and
GITTAGREGEX to UPSTREAM_CHECK_GITTAGREGEX to better reflect their purpose
and to reflect a common namespace.
(Bitbake rev: f0a9e783f9969573fd74edfa241ef14f18ac684e)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For two reasons:
1) The important one: we hit the following bug when doing upstream version checks
on some webpages:
https://bugs.launchpad.net/beautifulsoup/+bug/1471755
2) Also, documentation for beautifulsoup states that memory usage and
speed is improved that way.
(Bitbake rev: 7546d4aeb3ba8fda9832081b84d93138dc5e58d6)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This looks reasonable until you realise self.localpath is a function. Data
expansion of something which isn't a string is the original value so this
code just wastes CPU cycles and makes no sense. Remove it.
(Bitbake rev: 37214ea9bf484998b75dbc1200d53f1afc5257ed)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for the Subversion fetcher to checkout modules to a custom path
than the module name to avoid checkout is always module - svn is path
based and tag/branch-checkout might break builds because of invaid path specs.
(Bitbake rev: af88f538e61afa1b115be4d7afe00d8477f61750)
Signed-off-by: Jens Rehsack <sno@netbsd.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
errno.ENOENT checked if deletion of the fullmirror fails.
Exception was thrown since module not imported
(Bitbake rev: d92ebfc34b69ad5df2d151e6b8299fbb5afa3e5f)
Signed-off-by: Logan Buchy <logan.buchy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently this module is dereferencing errno.ENOENT but the python module "errno"
is not imported, which causes a crash when fetching from a git repository.
(Bitbake rev: 93e4c9bb2393b1074f5a01e7eaaac742a59d8086)
Signed-off-by: Romain Perier <romain.perier@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were maintaining state in the form of ud.repochanged to determine whether
we need to write the tarball in the case where the tarball already exists, but
this state is maintained only in memory. If we need to update the git repo,
set ud.repochanged to True, and then are interrupted or killed, the tarball
will then be out of date.
Rather than maintaining this state, simply remove the out of date tarball when
we update the git repo, and it will be re-created with updated content in
build_mirror_data.
[YOCTO #6366]
(Bitbake rev: eaaa81393f181432c8586b17ade623f42c9fed2e)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of reproducessing the same line over and over and over, we remove the
current line from the mirror list. This permits us to re-evaluate the list
while excluding all matches that have previousily occured.
Without this fix, adding this test results in a failure:
RuntimeError: maximum recursion depth exceeded in cmp
(Bitbake rev: 24a8e9a5b0ba145ae589178d74365c986ebca325)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When our clone exists, but is out of date, and the attempt to update it fails,
we don't necessarily want to remove the entire clone, particularly if it's
a large repository.
(Bitbake rev: 19af272ba5256653edeff6acbceeb09e3e478d61)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the main fetch method doesn't support checksums, the user will not be
defining them in the recipe, so we don't want to check them for
premirrors/mirrors either. This ensures that we never error due to missing
checksums on a git mirror tarball.
(Bitbake rev: 24c79bbed361b37f12d3351af13602e3d4386f4c)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using an absolute file URI, there's no host, and the path starts with
'/', the dir under ${DL_DIR}/git2/ ends up starting with '.', so is hidden.
Remove any leading '.' to fix this.
(Bitbake rev: 8dce6964d56b36a77fb113f2ad496cc992a5ff36)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With python you should not assign a list as the default value of a
function parameter - because a list is mutable, the result will be that
the first time a value is passed it will actually modify the default.
Reference:
http://docs.python-guide.org/en/latest/writing/gotchas/#mutable-default-arguments
(Bitbake rev: 7859f7388f2e3f675d0e1527cfde18625f36f637)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
[\.-_] means "any character between . and _"
What was meant here is certainly "any character from the three characters .-_"
(Bitbake rev: af13eaba627f199f91c048c435b9dbe19c79527f)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now latest_versionstring method returns (version, revision) for comply
the new return convention needed by SCM's like git get the current
revision.
bb/tests/fetch.py: Updated wget latest_versionstring test for comply new
convention.
[YOCTO #7605]
(Bitbake rev: 8d454646cbe1b04758ca178d8c6fcfd02b818b7b)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to know the revision associated with the upstream version in
SCM like git, this change broke return convention, oe-core recipeutils
get_recipe_upstream_version need to be updated.
tests/fetch.py: Updated git latest_versionstring test for comply new
convention.
[YOCTO #7605]
(Bitbake rev: fd175dc90024c503134c11cbd83e77d88c406ac8)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Don't limit the tag search for only tags end with ^{}.
(Bitbake rev: 7006ab313766344cf33481228465082ed5977d28)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since Python 2.7.9 ssl cert validation is enabled by default
see PEP-0476, this causes verification errors on some https
servers so disable by default.
(Bitbake rev: e177170200ece76b36e3f7d5597651fdef67736f)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Only use ProxyHandler opener when exists proxies in env.
(Bitbake rev: 0f062c5d99e12ce20c0e46a8e602448032144ff6)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2/__init__.py: Add connection_cache param in Fetch __init__.
In order to pass connection cache object to checkstatus method.
[YOCTO #7796]
(Bitbake rev: 9fa6407e6cefe66c77467419a8040d6957a6bb01)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Set BatchMode=yes instead of PasswordAuthentication=no. This will make
sftp fail immediately, not only when SSH requires interactive
authentication, but also on errors related to host key verifcation.
(Bitbake rev: 31305853a177735cc9c4553ea8905cd0acfcb100)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Create a branch and named as upstream branch when checkout source
* Set the branch to track remote branch.
(Bitbake rev: 1ba20e4fe9c884515b200589fe379ad5eeda10bd)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to pass connection cache object to checkstatus function
add fetch parameter.
(Bitbake rev: fbb9c6f5538084e125b58118a86968908e6f895b)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
FetchConnectionCache class acts as a container for socket connections
useful when implement connection cache into fetcher modules.
(Bitbake rev: 454da2cd17539ceb9caad6d76f034757e44ee12f)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is nessary when specified branch with submodules is different then
default (master) branch.
[YOCTO #7771]
(Bitbake rev: f7b0b5e33e00f3ce0744322eee93835ee76bf184)
Signed-off-by: Felipe F. Tonello <eu@felipetonello.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
* do_unpack error:
abort: repository DL_DIR/hg/vim.googlecode.com/hg/vim not found!
* The mirror tarball doesn't work
- Add the build_mirror_data to create the tarball
- Unpack the mirror tarball when needed
* The hg files will put in the dir like git: DL_DIR/hg, it was
DL_DIR/hg/path/to/src/uri/path in the past.
(Bitbake rev: 378647281ae883dd726f1e1b775dd35ef6a0e8d1)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fetching directories with p4 worked but single files did not. This
patch from Helmut Auer (helmut.auer@harman.com) fixes that issue.
[YOCTO #7891]
(Bitbake rev: 39da6579901c62a83ed9319c2016c58fbbc108fe)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than just use d.getVar(X), use the more explict d.getVar(X, False)
since at some point in the future, having the default of expansion would
be nice. This is the first step towards that.
This patch was mostly made using the command:
sed -e 's:\(getVar([^,()]*\)\s*):\1, False):g' -i `grep -ril getVar *`
(Bitbake rev: 659ef95c9b8aced3c4ded81c48bcc0fbde4d429f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Like the Git fetcher, the Mercurial fetcher shouldn't expect recipes to
provide a checksum. As described [1], recipes using a mercurial
fetcher that don't provide a checksum will fail in a the repository has
previously been downloaded and archived.
Credit to Rafaël Carré for figuring out the bug.
[1] https://www.mail-archive.com/openembedded-devel@lists.openembedded.org/msg41328.html
(Bitbake rev: 2df35a25b4968f64adfa673d5b73442c1a30829d)
Signed-off-by: Ash Charles <ashcharles@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
gitpkgv_revision returns a sortable revision number that can be used
in the PKGV variable for example. To mimic meta-openembedded gitpkgv
behaviour to provide a sortable revision numner, one could set the
following:
PKGV = "1.0+${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"
This would yield a package version like "1.0+69+fb5eb80".
(Bitbake rev: 989c08f62aff7b707c25c692c23284f16506b7bc)
Signed-off-by: Mike Looijmans <mike.looijmans@topic.nl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The idea here is to support package version numbering similar to gitpkgv in
meta-openembedded. This commit is the first step towards such functionality.
The original plan was to add a "get_pretty_srcrev" method to the fetcher, as
per Richard's suggestion [1]. While writing this, I noticed that it would
become a copy of get_srcrev with only two lines changed. So to create something
more Pythonic than a boolean argument and conditionals around the calls to the
fetcher's sortable_revision, I just made the method to be called on the fetcher
an argument to the method. Defaulting to 'sortable_revision' prevents affecting
existing code.
Now if the git fetcher were to implement, say 'gitpkgv_revision' one could
set the following in a recipe:
PKGV="1.2+${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"
and this would yield the same result as gitpkgv's GITPKGV variable.
See for the discussion leading to this change:
[1] http://lists.openembedded.org/pipermail/openembedded-core/2015-January/100345.html
(Bitbake rev: 2f1f4483493cc290f5d2c07f9906e90eaea2f4c1)
Signed-off-by: Mike Looijmans <mike.looijmans@topic.nl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Pass proper repository url without arguments after a semicolon.
Executing checkuri on a rule with git repository in SRC_URI does
not report errors when working offline because wrong repository
url is passed to the ls-remote command. For example
"bitbake -c checkuri glibc" command executes:
"git -c core.fsyncobjectfiles=0 ls-remote git://sourceware.org/git/glibc.git;branch=release/2.21/master"
command in a shell subprocess to determine if url is valid.
Shell subprocess executes in fact 2 commands:
"git -c core.fsyncobjectfiles=0 ls-remote git://sourceware.org/git/glibc.git"
and
"branch=release/2.21/master"
First one returns 127 or 128 depending on error but second one
returns 0 because it is just env variable setup. Therefore we're not catching
connection error.
[YOCTO #7558]
(Bitbake rev: efa44d04137977f883db4a643b0f774e91514722)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
First, when building mirror urls we don't do any fetching so we should never
be calling clean functions.
Currently, if a mirror url fails, we don't process it further to see
if there are any mirrors of the mirror.
We should do this even when the mirror url fails, else we may miss out
on valid/useful mappings, particularly in the case of file:// urls.
(Bitbake rev: b7fd3ec9994f664b17fc86423e6e7afac07e897b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow GIT_SMART_HTTP to be passed through to the fetch command so that
servers that cannot use GIT_SMART_HTTP can be used by the fetcher.
(Bitbake rev: e5c97a85bed0436d48eeaac2e32962cfb5371d2f)
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The URL is not sent when _latest_revision generates and exception.
When performing the sanity checks it is not possible to know the URI that failed.
This add the URL when latest_revision generates an exception.
[YOCTO: #7592]
(Bitbake rev: 9f2115b07a55cb14e4a74dc6fbd3707c28a234d0)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you're interested in using the checked out repository for development
(e.g. in OE with devtool) then you ideally want the origin remote to
point to the repository it was fetched from, so just set that after
cloning.
(As part of this I did a minor refactor so we have one function to
generate the repository URL, which was already in two places.)
Fixes [YOCTO #7756].
(Bitbake rev: 80ecd1c54d4c748cee3a7ce0d64013a346e7671e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BB_ALLOWED_NETWORKS is a list of hosts that the fetcher will be allowed
to use when BB_NO_NETWORK is not set.
If BB_NO_NETWORK is set, then networking is still disabled.
If BB_ALLOWED_NETWORKS is not set, the behaviour remains the same as
today.
If BB_NO_NETWORK is NOT set, and BB_ALLOWED_NETWORKS is configured, then
only the hosts in the list are usable by the fetcher.
eg:
BB_ALLOWED_NETWORKS="yoctoproject.org git.gnu.org"
The fetcher will be able to download from yoctoproject.org, git.gnu.org,
but not ftp.gnu.org or any other hostname that is not in the list.
There is also limited support for wildcards on the beginning of the
hosts, so BB_ALLOWED_NETWORKS="*.gnu.org" with match git.gnu.org and
ftp.gnu.org as well as foo.git.gnu.org
(Bitbake rev: c7263096ba31ba45daeeb9de90c1cb9ebef24a28)
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use hg clone and hg pull to copy the source into the build
directory rather than taring up the cloned repository and
untarring in the destination.
This allows submodules to be cloned. While here, make the default
behaviour keep the hg scm data to match the behaviour of the git
fetcher.
(Bitbake rev: f002b1ca80cb542a4ed0c06c53c914cd5e076565)
Signed-off-by: Daniel McGregor <daniel.mcgregor@vecima.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Setting BB_ENV_EXTRAWHITE to include GIT_SSL_CAINFO is enough to
get "git ls-remote" to work for recipe parsing but it is not enough
to get the fetcher to clone properly.
This is because the fetcher has its own idea about what variables
should be exported in the environment for some operations.
It is desirable to use alternate CA Bundles for internal testing prior
to using public keys for https, so we should allow the GIT_SSL_CAINFO
to pass through.
(Bitbake rev: 40ff92282bbf32cf644b021bf7cbb1f393dbb856)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Mariano Lopez
Maxin B. John
Mario Domenech Goulart
Enrico Scholz
Paul Eggleton
Ross Burton
Andrew Bradford
Stephano Cetola
Peter Kjellerstedt
Richard Purdie
Leonardo Sandoval
Robert Yang
Olof Johansson
Aníbal Limón
Derek Straka
Markus Lehtonen
Brendan Le Foll
Alexander Shashkevich
Aníbal Limón
Felipe F. Tonello
Kristian Amlie
Juro Bystricky
Christopher Larson
Alexander Kanavin
Jens Rehsack
Logan Buchy
Romain Perier
Mark Hatle
Ash Charles
Mike Looijmans
Marcin Smoczyński
Liam R. Howlett
Daniel McGregor
Jason Wessel