If there is a network failure the return value from latest_versionstring() is
('','') which later causes an exception when comparing versions.
Improve this by checking the return value and failing the test early.
[ YOCTO #12053 ]
(Bitbake rev: 4284d918dc7451f6a4d00be065369e86bbaec4d2)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if you use the subdir parameter in a SRC_URI and pass an absolute path
then it gets appended to the unpack directory instead of being used directly.
This is inconvenient as it may be useful to use ${S} when you want to unpack a
file into the source tree.
Change this behaviour so that absolute paths are used directly instead of being
appended to the root directory. To ensure that recipes cannot write files to an
arbitrary location enforce that the subdir starts with the unpack root.
(Bitbake rev: c3873346c6fa1021a1d63bddd9b898a77c618432)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Set this env variable to 'yes' to preserve temporary directories used by
the fetcher tests. Useful for debugging tests.
(Bitbake rev: 04132b261df9def3a0cff14c93c29b26ff906e8b)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The CUPS ipptool URL we were checking now redirects to github where the tarball
isn't present, so remove it from the test suite.
(Bitbake rev: 4b50895fb3462b21e3874a2e99c363c8d05e89e6)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this, the dict can reorder causing sanity test failures.
(Bitbake rev: ca8c91acc9396385834b266d4e8b84d917e5e298)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The tests were skipped when running without network even though they
didn't require network. This commit also adds a test case for URLs with
ports in them (the ports should not be considered when doing trusted
network checks).
(Bitbake rev: 77747de6b20538063eef3b188489a35bef225359)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update the upstream url used for testing cups versions after upstream website
changes.
(Bitbake rev: 5f06041d4936fc22297945bbbad7020bfa9083c6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When in SRC_URI appears file://dir;subdir=foo unpacker copies 'dir' to ${WORKDIR}, not
${WORKDIR}/foo as it should be.
These changes are fixing following bugs as well:
Bug 6128 - Incorrect wildcard unpack behaviour in fetcher
Bug 6129 - Local directories unpack to a different location than local files
(Bitbake rev: e659a3b0c2771679057ee3e13cd42e6c62383ff2)
Signed-off-by: Alexander Shashkevich <alex@stunpix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we specify urls such as npm://somehost;someparams the fetcher currently
does a poor job of handling mirrors of these urls due to deficiencies in the
way decodeurl works. This is because "somehost" is returned as a path, not
a host.
This tweaks the code so that unless its a file url, the host is returned
correctly.
This patch also adds test cases for these urls to the exist set of test
urls.
We need to tweak the URI() class since this thinks this is a relative url
which is clearly isn't. We also need to handle the fact that encodeurl will
error if passed a url of this form (it would want the path to be '/'.
(Bitbake rev: 83203cd2e677706e0111892a7843b83263cb8bd9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The core change here is to fall back to GET requests if HEAD is rejected in the
checkstatus() method, as you can't do a HEAD on Amazon S3 (used by Github
archives). This meant removing the monkey patch that the default method was GET
and adding a fixed redirect handler that doesn't reset to GET.
Also, change the way the opener is constructed from an if/elif cluster to a
conditionally constructed list.
(Bitbake rev: 6ec70d5d2e330b41b932b0a655b838a5f37df01e)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rename REGEX to UPSTREAM_CHECK_REGEX, REGEX_URI to UPSTREAM_CHECK_URI, and
GITTAGREGEX to UPSTREAM_CHECK_GITTAGREGEX to better reflect their purpose
and to reflect a common namespace.
(Bitbake rev: f0a9e783f9969573fd74edfa241ef14f18ac684e)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of reproducessing the same line over and over and over, we remove the
current line from the mirror list. This permits us to re-evaluate the list
while excluding all matches that have previousily occured.
Without this fix, adding this test results in a failure:
RuntimeError: maximum recursion depth exceeded in cmp
(Bitbake rev: 24a8e9a5b0ba145ae589178d74365c986ebca325)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Change the test name to be more specific on what is tested.
(Bitbake rev: 3e39156bc330c4c726058a5b9c13d33e2daad89f)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now latest_versionstring method returns (version, revision) for comply
the new return convention needed by SCM's like git get the current
revision.
bb/tests/fetch.py: Updated wget latest_versionstring test for comply new
convention.
[YOCTO #7605]
(Bitbake rev: 8d454646cbe1b04758ca178d8c6fcfd02b818b7b)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to know the revision associated with the upstream version in
SCM like git, this change broke return convention, oe-core recipeutils
get_recipe_upstream_version need to be updated.
tests/fetch.py: Updated git latest_versionstring test for comply new
convention.
[YOCTO #7605]
(Bitbake rev: fd175dc90024c503134c11cbd83e77d88c406ac8)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add some tests of mirrors or mirrors to the fetcher unittests.
(Bitbake rev: e33d82bc10283d533f928836d56a6f0af80ea5c1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fetcher calls os.chdir() in a number of places, which can affect
other tests (since the directory it changes into gets deleted) - let's
just put the current directory back to where it was when we're done.
(This fixes bb.tests.Path.test_unsafe_delete_path failing if it was run
as part of a full bitbake-selftest run, where the fetcher tests get to
run before it.)
(Bitbake rev: b1653855c74f86909c9f329ed6d2b10391c28395)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BB_ALLOWED_NETWORKS is a list of hosts that the fetcher will be allowed
to use when BB_NO_NETWORK is not set.
If BB_NO_NETWORK is set, then networking is still disabled.
If BB_ALLOWED_NETWORKS is not set, the behaviour remains the same as
today.
If BB_NO_NETWORK is NOT set, and BB_ALLOWED_NETWORKS is configured, then
only the hosts in the list are usable by the fetcher.
eg:
BB_ALLOWED_NETWORKS="yoctoproject.org git.gnu.org"
The fetcher will be able to download from yoctoproject.org, git.gnu.org,
but not ftp.gnu.org or any other hostname that is not in the list.
There is also limited support for wildcards on the beginning of the
hosts, so BB_ALLOWED_NETWORKS="*.gnu.org" with match git.gnu.org and
ftp.gnu.org as well as foo.git.gnu.org
(Bitbake rev: c7263096ba31ba45daeeb9de90c1cb9ebef24a28)
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add set of PN in data because now latest_versionstring use it for
validate version directory searching.
(Bitbake rev: 2e4a03db967ac1459b2764108fc54c4566a7e371)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The versionstring tests hit the network so should only run when
network tests are enabled.
Also remove the print statement which confuses the test output and
add it to the test failure message instead.
(Bitbake rev: 6c046660cfc4fb3792a42aeafff91a13f68a2e89)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update test case for cups is needed because match only 2.0.0
versions see VERSION=2\.0\.0 in the previous string.
(Bitbake rev: 148aba30155f4de17f6c6fb9b4c37f08a2db202b)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Being able to query whether updated versions of a url are available
is useful, not least for the package reporting system. Since such code
is closely linked to the url type and the url itself, the fetcher
makes a locical place to contain this code.
For wget based urls this means taking upstream directory listings
and searching those for later versions, returning those that are
found.
The patch also adds unittests for this function so that if
improvements are made, the original test urls can be used
to evaulate the those changes.
This is based on code from Irina Patru <irina.patru@intel.com>.
(Bitbake rev: a8272e22b7819e0e8afd8e291d276f5f28fc0007)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Being able to generate a version string representing the most recent git commit
given git is useful, not least for the package reporting system.
This adds in a latest_versionstring method to the git fetcher
which allows users to query the latest version using ls-remote
and filtering the responses.
The patch also adds unittests for this function so that if
improvements are made, the original test urls can be used
to evaulate the those changes.
This is based on code from Irina Patru <irina.patru@intel.com>.
(Bitbake rev: f71c8c0354e87fed80bc845db6728e6e18ce9c4d)
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fixes urls of the form file://some/path/file;subdir=b. It also
adds in a couple of tests so we now tests these corner cases.
(Bitbake rev: 46306912a96444790efa9418d934dfdd36773ba1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add some explicit tests for unpacking local files to the appropriate
location. Some of these tests are actually testing for broken behaviour;
these have been called out in the comments, and associated bugs have
been filed.
(Bitbake rev: ca921c773c52392a5a338b2f493ad38c8132f708)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When setting the username after already having set the password, the password
was unexpectedly reset. This change fixes this issue and introduces unit tests
to make sure it doesn't happen again.
(Bitbake rev: 25faef3a047f9c7564089463d7c96f6910b640cb)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a case in meta-intel where a SRC_URI contains both a query string and
URI parameter:
https://edc.intel.com/Download.aspx?id=6190;downloadfilename=LIN_IEMGD_1_14_GOLD_2443.tgz
Python's urlparse thought the URI parameters were part of the query parameter
value, but in the bitbake context this is obviously not the case. As bitbake's
usage isn't really RFC compliant, we have to extract and remove the URI parameters
*before* urlparse sees the URI.
(Bitbake rev: c2f27ae4271985b48f957747b6ea372c8699cb49)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This change introduces the .query property of the URI class. It is a
read/write dict of the parameters supplied in the query string of the
URI. E.g.:
http://example.com/?foo=bar => .query = {'foo': 'bar'}
(Bitbake rev: 1cb2b3c458c8c5521591d2c8f2e0058143fc77bb)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use a newly created "git-submodule-test" repo on git.yoctoproject.org
which currently contains one submodule (the bitbake repository).
(Bitbake rev: a750c57242928c546a5aace632543e956ee908eb)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a sanity check so that if some SRCREV is set and a rev parameter is given
to the url, the revision given should match.
Any tag parameter behaves the same as rev. If both are specified, error to
tell the user we're confused rather than do something which may or may not
be what they intended.
Also add some unittests for this.
(Bitbake rev: e82a4ab48991035866da9914c8b75a9bfbc9a7fc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* currently decode_url regexp parses branch=@foo as username so it ends like this:
- ('git', '', 'foo', 'git.openembedded.org/bitbake;branch=', '', {})
+ ('git', 'git.openembedded.org', '/bitbake', '', '', {'branch': '@foo'})
* http://hg.python.org/cpython/file/2.7/Lib/urlparse.py also assumes
that there is at least one '/' as separator between netloc and path,
params, so it looks reasonable to prevent including '/' in username
(Bitbake rev: 2c82742114091cb55055328b54223686816582f2)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The skipIf decorator is unavailable under python 2.6 and finding a syntax
works there is hard. This patch does allow network tests under 2.6.
(Bitbake rev: 565b3e31e0226c6e193ee0b031bd5e7bef25591e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the fetch tests fail on python 2.6 since check_output
doesn't exist on this version of python. Use bb.process instead
to avoid this problem.
(Bitbake rev: a3637155e829e06bbfe50fb8753de3de7ee8c22e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With this change, you can opt-out to skip unit tests that require an
internet connection. To opt-out, you'll set the environment variable
BB_SKIP_NETTESTS to 'yes'.
(Bitbake rev: 9ff5f172096a4f51b6b085307506473405dc4f59)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When writing mirror specifications, the current regexp syntax can be awkward
and hard to get it to do what you want. For example, extracting the 'basename'
of a repository:
PREMIRRORS = "git://.*/([^/]+/)*([^/]*) git://somewhere.org/somedir/\\2;protocol=file"
can now become:
PREMIRRORS = "git://.*/.* git://somewhere.org/somedir/BASENAME;protocol=file"
which is much clearer. A MIRRORNAME substitution is also added which contains
an encoded form of both host and path. One of the problems with the existing
regexp syntax is you couldn't access HOST information from PATH and vice-versa
which is an issue this patch also addresses.
Tests for the new syntax are also added.
(Bitbake rev: c6b1acbad7b3d2698530eb8b5249adb4ab95da21)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ross Burton
Markus Lehtonen
Richard Purdie
Olof Johansson
Alexander Shashkevich
Alexander Kanavin
Mark Hatle
Aníbal Limón
Aníbal Limón
Paul Eggleton
Liam R. Howlett
Martin Jansa