If we're fetching outside of the context of a recipe, it's handy to be
able to disable checksum functionality so you don't get a meaningless
warning about the signatures being missing.
(Bitbake rev: 49dbcfbc56a206964acc5de761bba31be0283ba1)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
See http://www.nongnu.org/lzip/lzip.html for details on the compression
format.
(Bitbake rev: 9027b1273b5405c7269b013604ab417771b5eafe)
Signed-off-by: Koen Kooi <koen@dominion.thruhere.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Forcing strict to be a string, to avoid problems when performing comparisons
[YOCTO #6762]
(Bitbake rev: b8ed2098bdea2afd93ab4e3e1b834f3a31cb60de)
Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fixes urls of the form file://some/path/file;subdir=b. It also
adds in a couple of tests so we now tests these corner cases.
(Bitbake rev: 46306912a96444790efa9418d934dfdd36773ba1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Check if the 'subdir' parameter exists and assign it to 'destdir' so that
files are copied in ${WORKDIR}/destdir. This fixes urls that are of the form
file://a;subdir=b.
(Bitbake rev: 836a986b365eb9798563ec08d90b346596de7791)
Signed-off-by: Roxana Ciobanu <roxana.ciobanu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The clearcase fetcher is used to retrieve files from a ClearCase
(http://en.wikipedia.org/wiki/Rational_ClearCase) repository.
Usage in the recipe:
SRC_URI = "ccrc://cc.example.org/ccrc;vob=/example_vob;module=/example_module"
SRCREV = "EXAMPLE_CLEARCASE_TAG"
PV = "${@d.getVar("SRCREV").replace("/", "+")}"
The fetcher uses the rcleartool or cleartool remote client, depending on which one is available.
Supported SRC_URI options are:
- vob
(required) The name of the clearcase VOB (with prepending "/")
- module
The module in the selected VOB (with prepending "/")
The module and vob parameters are combined to create
the following load rule in the view config spec:
load <vob><module>
- proto
http or https
Related variables:
CCASE_CUSTOM_CONFIG_SPEC
Write a config spec to this variable in your recipe to use it instead
of the default config spec generated by this fetcher.
Please note that the SRCREV loses its functionality if you specify
this variable. SRCREV is still used to label the archive after a fetch,
but it doesn't define what's fetched.
User credentials:
When using cleartool:
The login of cleartool is handled by the system. No special steps needed.
When using rcleartool:
In order to use rcleartool with authenticated users an `rcleartool login` is
necessary before using the fetcher.
(Bitbake rev: 76cff0aaad4cf10b325bceadd8ea90b3d75edcc2)
Signed-off-by: Dennis Meier <meier.dennis@siemens.com>
Reviewed-by: Roger Meier <r.meier@siemens.com>
Reviewed-by: Christian Liechti <christian.liechti@siemens.com>
Reviewed-by: Henrique Mendonca <henrique.mendonca@siemens.com>
Reviewed-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Handle the following situations in a URL (e.g. in SRC_URI):
* Trailing semicolon in a URL - this is now ignored.
* Parameter specified with no value (no equals sign). This still
produces an error, but at least it is MalformedUrl with a proper
message rather than "ValueError: need more than 1 value to unpack".
(Bitbake rev: bfd13dfbc4c9f1dd8315002271791b1d9e274989)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 42b7be1a6c5ff559f4a213eed04b370ca955460d)
Signed-off-by: Nick Lewis <nick.lewis@usa.g4s.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The initial version of this patch was reverted in
0a94e568152de550dedc8135a766beb18bf064ab
However, it was later agreed upon that it was reverted due to
a misunderstanding during code review; the patch is ok.
This version of the patch also removes an unneeded initialization of
the 'checksum' variable outside the scopes it is being used in.
(Bitbake rev: 9ee19a3ca2f8e11a91f0289ea3486310c61d40f2)
Signed-off-by: Jacob Kroon <jacob.kroon@mikrodidakt.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit 6dfb33fd7204e09815bde7ba5bf21941a73fe965.
It causes an unindeded change in behaviour.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Cleanup the fix done in f9416e76e272ba3249abff099f6f3a47fe82e03e.
Instead of adding continue statements we can just move the last
statements into the final else-clause.
(Bitbake rev: 6dfb33fd7204e09815bde7ba5bf21941a73fe965)
Signed-off-by: Jacob Kroon <jacob.kroon@mikrodidakt.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there is a directory matching a wildcard in SRC_URI when getting file
checksums, we should recurse into that instead of producing an error.
(Bitbake rev: ae87b7eb414e3d5eefd2effec7b30c22d2186b02)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The Local fetcher's localpath is returning the parent directory for a
wildcard match; we need to handle this and add the wildcard
specification so that we checksum the correct files.
Fixes [YOCTO #6127].
(Bitbake rev: 4a90edd62c16cbf41b5b93280e155077564c774a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously, if a file listed in SRC_URI was missing at the time of
parsing, and then was added and bitbake run again, it would not be
picked up because the recipe was cached without it. If we allow the file
to be added to the list of files to checksum, then it will be checked
for and found on the second run.
Fixes [YOCTO #4790].
(Bitbake rev: 71da822762cb298261cccdfa54b9c0fea02c3c5d)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For a while its been puzzling me why connman-gnome rebuilds as often as it
does. It turns out you can trigger this with a new checkout of the metadata.
The SRC_URI that is causing the problems is:
SRC_URI = "file://images/*"
and rather oddly the results in checksums for a file "." being added to
the tree, e.g.:
('.', 'ab48a68186f0e0f277c21ef4cb390b4b')
The problem is that when iterating files lists, the checksum variable can
become set yet we don't break the out from the for loop, which leads to
odd (and non-deterministic) entries being added into the file checksum list.
The exact item added probably depends on the order of items on the disk.
Before this change, bitbake-diffsigs on connman-gnome:do_fetch would report:
This task depends on the checksums of files: [
('connman-signal-03.png', 'f6c16aee57b37b73793a2f1dea433ffa'),
('connman-signal-02.png', 'ad0cd22710c097d8174121fc1023c3be'),
('connman-signal-01.png', '8842bd83d2fa9ba56480df34c727c629'),
('null_check_for_ipv4_config.patch', 'a23271e41c9fe81551244d875106af10'),
('connman-signal-05.png', '808589e7e8d502b44c7b007e9e68d48c'),
('connman-signal-04.png', 'ab48a68186f0e0f277c21ef4cb390b4b'),
('null_check_for_ipv4_config.patch', 'a23271e41c9fe81551244d875106af10'),
('0001-Removed-icon-from-connman-gnome-about-applet.patch', 'e2d8269357c1e8c84291606da24eea85'),
('0001-Removed-icon-from-connman-gnome-about-applet.patch', 'e2d8269357c1e8c84291606da24eea85'),
('.', 'ab48a68186f0e0f277c21ef4cb390b4b')]
Afterwards:
This task depends on the checksums of files: [
('connman-signal-03.png', 'f6c16aee57b37b73793a2f1dea433ffa'),
('connman-signal-02.png', 'ad0cd22710c097d8174121fc1023c3be'),
('connman-signal-01.png', '8842bd83d2fa9ba56480df34c727c629'),
('null_check_for_ipv4_config.patch', 'a23271e41c9fe81551244d875106af10'),
('connman-signal-05.png', '808589e7e8d502b44c7b007e9e68d48c'),
('connman-signal-04.png', 'ab48a68186f0e0f277c21ef4cb390b4b'),
('null_check_for_ipv4_config.patch', 'a23271e41c9fe81551244d875106af10'),
('0001-Removed-icon-from-connman-gnome-about-applet.patch', 'e2d8269357c1e8c84291606da24eea85'),
('0001-Removed-icon-from-connman-gnome-about-applet.patch', 'e2d8269357c1e8c84291606da24eea85')]
which is correct and deterministic without the "." entry.
(Bitbake rev: f9416e76e272ba3249abff099f6f3a47fe82e03e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When setting the username after already having set the password, the password
was unexpectedly reset. This change fixes this issue and introduces unit tests
to make sure it doesn't happen again.
(Bitbake rev: 25faef3a047f9c7564089463d7c96f6910b640cb)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a case in meta-intel where a SRC_URI contains both a query string and
URI parameter:
https://edc.intel.com/Download.aspx?id=6190;downloadfilename=LIN_IEMGD_1_14_GOLD_2443.tgz
Python's urlparse thought the URI parameters were part of the query parameter
value, but in the bitbake context this is obviously not the case. As bitbake's
usage isn't really RFC compliant, we have to extract and remove the URI parameters
*before* urlparse sees the URI.
(Bitbake rev: c2f27ae4271985b48f957747b6ea372c8699cb49)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This change introduces the .query property of the URI class. It is a
read/write dict of the parameters supplied in the query string of the
URI. E.g.:
http://example.com/?foo=bar => .query = {'foo': 'bar'}
(Bitbake rev: 1cb2b3c458c8c5521591d2c8f2e0058143fc77bb)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The svk fetcher never appears to be used by anyone and the development
on svk appears to have stopped in 2010. We might as well drop support
for it.
(Bitbake rev: 8239264753977bd06ad5b1b574245d3842af489b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A typo was meaning that the mirror creation method wasn't being called
when it should have been. Fix the type to fix mirror tarball creation.
[YOCTO #5284]
(Bitbake rev: 348971d410bfd5d8b1757468d73e1d24ae78a594)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current message can be ambiguous, improve it (and also rename a
variable to clean up the rest of the function).
(Bitbake rev: 0c1bb7c0fce7b0f334311a2893ccb00385fa8d55)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a sanity check so that if some SRCREV is set and a rev parameter is given
to the url, the revision given should match.
Any tag parameter behaves the same as rev. If both are specified, error to
tell the user we're confused rather than do something which may or may not
be what they intended.
Also add some unittests for this.
(Bitbake rev: e82a4ab48991035866da9914c8b75a9bfbc9a7fc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently INVALID and None are checked as incorrect values under different
circumstances. This code standardises those checks to be consistent. We
should phase out the use of "INVALID".
(Bitbake rev: 86ef4e65ce18b71dc69643586bd2aa8f48703171)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fetcher will try:
1) PREMIRROR
2) Upstream
3) MIRROR
If it fails to download from the Upstream, but succeeds from the MIRROR,
and ud.localpath != origud.localpath (for example, the git tarball),
then we will get the error (e.g.: xf86-video-omapfb):
ERROR: Function failed: Fetcher failure for URL: 'xxx'. Unable to fetch URL from any source.
ERROR: Logfile of failure stored in: /path/to/log.do_fetch.28024
It should not show the error and let the build go on since it succeeds.
(e.g.: xf86-video-omapfb)
[YOCTO #5686]
(Bitbake rev: c08ca1e4eeb04f78e1354780cf5a4c3855e49572)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* currently decode_url regexp parses branch=@foo as username so it ends like this:
- ('git', '', 'foo', 'git.openembedded.org/bitbake;branch=', '', {})
+ ('git', 'git.openembedded.org', '/bitbake', '', '', {'branch': '@foo'})
* http://hg.python.org/cpython/file/2.7/Lib/urlparse.py also assumes
that there is at least one '/' as separator between netloc and path,
params, so it looks reasonable to prevent including '/' in username
(Bitbake rev: 2c82742114091cb55055328b54223686816582f2)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This add a Git Annex backend which reuses the Git fetcher code; it
allows managing files with git, without checking the file contents
into git, being useful when dealing with files larger than git can
currently easily handle, whether due to limitations in memory, time,
or disk space.
(Bitbake rev: a61fc4db598e9d13c966712a6a0e4783e19448be)
Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was a problem:
$ bitbake xf86-video-omapfb -cfetch && bitbake xf86-video-omapfb -ccleanall
Everything should be removed, but the
0006-omapfb-port-to-new-xserver-video-API.patch.done still exists in the
DL_DIR, this is because the clean() in the fetch2/__init__.py skips
removing the local file, so that it will skip removing the .done.
The local file (file://) isn't needed to be removed since it is not
downloaded into DL_DIR, but the .done should be removed, this patch will
remove the .done, and it doesn't remove anything else since the clean()
in local.py does nothing.
[YOCTO #5687]
(Bitbake rev: 2bc99b9dfa532430a13c39fca4e5ef3a2206b3b8)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A part of fetch2/__init__.py uses 3 spaces as the indent, I
think that they should be typos.
(Bitbake rev: abafd85e2fcf23cee872e0e9e468898101430f1f)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Because of the way we were handling this error, it was printed twice -
once via logger.error() (to avoid the log being printed) and a second
time when the exception gets wrapped in a FuncFailed at a higher level.
Call logger.error() earlier and change the text we send in the
exception to be more brief, so it more closely resembles the behaviour
when there is an invalid checksum.
(Bitbake rev: 46765369d7f76ec7f67b90430131a79eb6a66235)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were checking SRC_URI md5sum/sha256sum values against None here, so
if they were set to "" then no error was produced. Since the value is
still effectively unset in this case, this is not the right behaviour;
just check if the value doesn't evaluate to False instead.
(Bitbake rev: 040943a718795c64dc4e604abfcf08b26b7d00e6)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is no good reason to keep passing around the url parameter when
its contained within urldata (ud). This is left around due to
legacy reasons, some functions take it, some don't and its time
to cleanup.
This is fetcher internal API, there are a tiny number of external users
of the internal API (buildhistory and distrodata) which can be fixed up
after this change.
(Bitbake rev: 6a48474de9505a3700863f31839a7c53c5e18a8d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
urldata contains the url so we might as well stop passing around
pointless function parameters. This was done for legacy reasons but
its time to clean this mess up.
This is a first step in cleanup and is a standalone patch but there is
more to be done in a second patch.
(Bitbake rev: 06590cfebbcf6565a17b80cc298e3ecdfaba4656)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If an SCM mirror is in PREMIRRORS, the tarball is downloaded and then found
by the "upstream" check and handled correctly.
If an SCM mirror is in MIRRORS, the tarball is downloaded but not used
since there is no "upstream" run after MIRRORS completes. It therefore
sits there useless and unused. This code change forces the upstream to
run after a mirror tarball is found and fixes the usage of SCM mirrors
in MIRRORS.
(Bitbake rev: a66ee0994645aa5658b2f5ea134ed17d89f8751a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a file ends with .xz, it currently gets overwritten during unpack:
The decompress command for .xz files is:
'xz -dc %s > %s' % (file, efile)
and as efile == file, we end up overwriting file (the source).
Fix this by adding .xz to the list of suffixes that that need to
be removed from a file name for an extract command, leaving the
bare file name. Now, for a given file foo.xz,
file == foo.xz and efile == foo, similar to how .gz .bz2 and .Z
files are treated.
(Bitbake rev: 2cd2d0a48e12ab4358fb967eaf7a56c17993f48d)
Signed-off-by: André Draszik <andre.draszik@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BB_NO_NETWORK can be set by bitbake internally by the use of
BB_FETCH_PREMIRRORONLY so update the error message to give users a
hint about this.
[YOCTO #3222]
(Bitbake rev: cac3060d0bf8c7deeacda18d06d92787911380d0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
AUTOINC was meant to appear once at the start of the version string.
The list of names may not be sorted meaning it could get inserted in
the middle. This patch simplifies the code and ensures it appears at
the start.
Include cache version bump to ensure the cache picks up these changes.
(Bitbake rev: ad8bf10d873abb94d987860a3f6d06b134fb8a99)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The assignment to True was missing from the code, well spotted Saul!
(Bitbake rev: e493fe8cb4953935f01361ffc0240e5818ebb283)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The long srcrevs are mainly used or the workdir construction as well as
the package version. The long entries are hashes generated by the git fetcher
and other scms using a similar revision mechanism.
We need these to change when the package changes however collisions are
unlikely to happen within the domains we care about. The long revisions
have generated negative user feedback due to the use in path and file
names.
This patch therefore truncates the revisions to 10 characters maximum.
This should be safe in the contexts where these revisions are used as
the chances of spatially close collisions is very low (distant
collisions are not a major issue in the way we use these).
(Bitbake rev: 43a8319cda7fae37862dae323eeb24cb39ca21b7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now we no longer try and provide increasing values from the fetcher,
we can simplify the function structure for the sortable_revision
pieces and move the AUTOINC handling directly into the function
which needs it, simplifying the code.
(Bitbake rev: fb068bee47bb1a06f02447daf16c2b2a79c03288)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move various random imports to the start of the modules as cleanup
and avoid an import issue with bb.process on python 2.6.
(Bitbake rev: aed4adfbe3a591ca4f8e41fb763c9f961bf2e6d5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* this can be useful when someone wan't to compare old file with
bad checksum and new one
(Bitbake rev: 33c6b93597dd43ab03ce7b62ba3eeb1893a68c38)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds very basic git submodule support to the fetcher. It can be
used by replacing a git:// url prefix with a gitsm:// prefix, otherwise
behaviour is the same as the git fetcher. Whilst this code should be
functional, its not as efficient as the usual git fetcher due to the
need to checkout the tree to fetch/update the submodule information. git
doesn't support submodule operations on the bare clones the standard git
fetcher uses which is also problematic.
This code does however give a starting point to people wanting to use
submodules.
(Bitbake rev: 25e0b0bc50114f1fbf955de23cc0c96f5f7a41e3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit 21fe2683aefde10e847e66c11c26d4f4c1e07cfd
since bitbake-selftest doesn't pass when this is applied and
we're seeing multiple build failures from this change.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fetcher differs from the SSH fetcher in that it adheres more
strictly to the SECSH URI internet draft --- it uses the sftp://
instead of the ssh:// scheme, and it uses sftp instead of scp.
(Bitbake rev: d240baeb7a4107d2eba3f08c411c0f086674d8e2)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A class representing a generic URI, with methods for accessing the URI
components, and stringifies to the URI. This class should be a bit more
flexible than the existing {encode,decode}_url functions in that it
supports more components (e.g. port) and that it does not rely on a
specific order on the return values. This makes it easy to add new
properties without affecting the API.
(Bitbake rev: bd824da8a7eafe27310e410807319628378caeca)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently stamps end up at the top level of DL_DIR even if the files
themselves are in a directory structure. This patch preserves path
components allowing the top level directory to be less populated
which is an advantage for large sstate caches.
(Bitbake rev: 59921ce3ed7a4c0b7f8ef1a101ad9127469bf1fd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For example, this allows us to specify downloadfilename=PATH in sstate
mirror urls to improve directory structures in DL_DIR when using sstate.
(Bitbake rev: 7850a1364b6b37c58664d84f9c14806b4479b45c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The following variables perform no function outside of bitbake:
GIT_CONFIG
GIT_PROXY_HOST
GIT_PROXY_PORT
GIT_PROXY_IGNORE
GIT_CONFIG only affects the git-config command which is not relevant to
the fetcher. This was previously used with the OE GIT_CORE_CONFIG
variable which would provide a basic git config to use instead of the
user's config. This usage was deprecated by git for over a year now:
http://git.661346.n2.nabble.com/Overriding-gitconfig-using-GIT-CONFIG-td6680977.html
GIT_PROXY_HOST and GIT_PROXY_PORT are not used by git.
GIT_PROXY_IGNORE was an OE construct used to create the custom git
config and had no meaning outside of the OE environment. It is not used
by git.
Remove these variables from the fetcher environment.
Users wishing to configure git to work with a proxy should define the
GIT_PROXY_COMMAND environment variable to use an external script.
NO_PROXY can be used within this script to skip the proxy for certain
hosts.
(Bitbake rev: e60270bdce6b8c2f8da1a4838aa374da9db3c86a)
Signed-off-by: Darren Hart <dvhart@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Applications are inconsistent in their use of upper and lower case proxy
variables. Curl, for example, specifies NO_PROXY (not no_proxy) in the
man page (changed in 2009 [1]). Avoid proxy issues by ensuring both the
upper and lower case versions of each proxy variable is available in the
environment for the fetcher commands.
Add FTPS_PROXY and ftps_proxy to the list as well.
1. http://curl.haxx.se/mail/tracker-2009-04/0012.html
(Bitbake rev: c3e6b2c5ec81d5ad7dcf606fff16fd5552bd267c)
Signed-off-by: Darren Hart <dvhart@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a particular SRCREV (say for a particular branch) is missing, the
fetcher will currently just report an obtuse error about the "SRCREV"
being invalid. If there is more information is to be had (say from name,
i.e. branch, and pn) then display that as well.
The new error looks something like this:
ERROR: ExpansionError during parsing /home/dvhart/source/poky/meta/recipes-kernel/linux/linux-yocto_3.4.bb: Failure expanding variable do_patch: ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure for URL: 'git://otcgit.jf.intel.com/dvhart/linux-yocto-minnow-3.4.git;protocol=git;nocheckout=1;branch=standard/minnow,meta,emgd-1.14;name=machine,meta,emgd'. Please set SRCREV_emgd_pn-linux-yocto to a valid value
Note the variable listed as invalid is
"SRCREV_emgd_pn-linux-yocto", making it explicit what is wrong.
(Bitbake rev: 63774f5b4edb999300bddd891233f6050f4af877)
Signed-off-by: Darren Hart <dvhart@linux.intel.com>
Cc: bitbake-devel@lists.openembedded.org
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Changing the path to a file could change the task hash even if the file
still has the same checksum. This occurs when the task depends on
multiple files and the sort order of their paths changes. Usually the
sorting is consistent because layers tend to have the same relative
paths, but this should take care of other configuations.
The problem arose when using a .bbappend to add files to a recipe in
another layer. If the layer is located alongside the other layers and
their parent directory is moved, the hash does not change. However,
moving the .bbappend layer outside of the common directory can change
the path sort order and the task hash.
(Bitbake rev: 22bd19d208f0251f5a1f9b98f3cac66181f3fc07)
Signed-off-by: Tyler Hall <tylerwhall@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- do not use the BB_URI_LOCALCOUNT database for computing revision
incremental numbers anymore
- sortable_revision now generates "AUTOINC+${latest_rev}"
- use one incrementing value rather than several
- PV becomes 0.1+gitAUTOINC+deadbeefdecafbad_decafbaddeadbeef
- remove all localcount code and simplify the fetcher
- this patch addresses the following proposal:
http://lists.linuxtogo.org/pipermail/bitbake-devel/2012-November/003878.html
(Bitbake rev: 61cf01c5c236b4218f40cfae7c059c2b86765dbd)
Signed-off-by: Constantin Musca <constantinx.musca@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently there is code which uses FILESDIR in unpack to ensure
parent directories are created, leading to differing behaviour depending on
which search path is used to locate the directory.
This change standardises the code and takes the data from the fetcher in
question meaning we can standardise the code and deprecate FILESDIR.
(Bitbake rev: 1cccb3bd01ed82e4978acfef0fda1bd797eef72a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add "-d" option to cpio since it is useful:
-d
--make-directories
Create leading directories where needed.
[YOCTO #3137]
(Bitbake rev: a78f9ded7896432b107f34c0bb608b389fdb676a)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Don't print the full exception in the initial warning - if we later
succeed in fetching the file from a mirror, we won't usually need the
details (which are in the fetch log if they are needed); otherwise the
full error will be printed when the fetch operation fails. Also adjust
the conditional block so that we don't print another warning just
mentioning we're going to try mirrors.
* Call logger.error() so that with knotty the full log is not printed
* Provide an explanation around the lines we print for easily updating
the checksums in the recipe. We don't want users to be just blindly
updating the recipe in case of a transient failure or deliberately
altered remote file.
(Bitbake rev: 2793413106c925b06783beb7413aa87cbcf246c3)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Most of the time we don't need to see the fetch command; the fetch log
includes the command as a debug message in any case, so omit it. Also
adjust the way command output is printed (we don't need stderr/stdout
labelled, and print "no output" instead of "output:\nNone" when there is
no output.
(Bitbake rev: a75505a52e4da918222100221f79e8a658f90446)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The implementation of NoMethodError and MalformedUrl was broken - if you
just set self.args in an exception class to a string it treats it as a
list and then fails later on with a TypeError due to the number of
arguments not matching up.
This nasty exception during exception handling was breaking the normal
exception flow (fixed separately), which meant that if you had a
malformed URL or invalid protocol in SRC_URI you would get the
following:
ERROR: Command execution failed: Traceback (most recent call last):
File "/home/user/poky/poky/bitbake/lib/bb/command.py", line 84, in runAsyncCommand
self.cooker.updateCache()
File "/home/user/poky/poky/bitbake/lib/bb/cooker.py", line 1207, in updateCache
if not self.parser.parse_next():
File "/home/user/poky/poky/bitbake/lib/bb/cooker.py", line 1694, in parse_next
logger.error('Unable to parse %s', value.recipe,
AttributeError: 'exceptions.TypeError' object has no attribute 'recipe'
A specific fix for [YOCTO #2977].
(Bitbake rev: 9d4150d99051d24ff218e8a43664ceaf524b19c7)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This ensures that if all a MIRRORS entry does is add a slash, this does
not result in a circular loop.
Fixes [YOCTO #3073].
(Bitbake rev: 57055d337a2c9997a6e5d5bdabaec396e3e128e9)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Unpack the ".rpm" binary package (only .src.rpm in the past)
* Unpack the .deb and .ipk binary package, their unpack commands are the same.
* This is useful for binary package recipe.
[YOCTO #1592]
(Bitbake rev: de7ceb9459574f33920ccc06255b533434f0ec25)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* in form which can be copied into a recipe without modification
* like oe-classic did since:
http://git.openembedded.org/openembedded/commit/?id=68abc465559a68e9201c9525be3c4acc6297eaed
* it shows them in right form when they are missing completely, but in
more verbose form when different
* it needs to print that only when checksums were requested, e.g.
fetching from sstate mirror sets both md5mismatch and sha256mismatch,
but your checksums shouldn't be shown
(Bitbake rev: 9eb34fc866775fd8310759a0111f232a9dc98981)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a file:// mirror is being used, the fetcher will create a symlink to the
local file. However, if the local file gets removed, that link will be dead,
and os.path.exists() returns False in that case, so it tries and fails to
recreate the link. Now we unlink such a dead link if it exists.
(Bitbake rev: 229ed3857e826e3e215e843cb51f729c1e13ed37)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without the new exception, when the system is configured to use premirrors,
but not allow network access (via BB_NO_NETWORK), when a recipe was lacking a
checksum the wrong error message(s) were being generated.
Instead of complaining about trying to perform network access, if the system
was able to find the item in the premirror, it should inform the user of the
lack of checksums, and the two SRC_URI fields they should use to update their
recipe.
(Bitbake rev: cb10e9c03a3f96d94e27e18330009616dde5e2b3)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As well, if "proto" is used, get the associated value as "protocol"
(Bitbake rev: 53e6b630f0463d2d07cdaa9c9eb36794dc9b6b69)
Signed-off-by: Andrei Gherzan <andrei@gherzan.ro>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
People are using regexps in the url type field so we need to preserve
this bitbake behaviour. To address the issues with https:// urls mapping
badly to file:// urls we anchor the regexp if its not already anchored.
There should be no expressions in the wild which would break with this
change.
(Bitbake rev: ce0579dc256251e523c6330641f98b9f5a0e5761)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If '*' does end up in mirror urls accidently, some strange things
can break since supports_checksum() looks for this, ud.localpath can
then get ignored and this can lead to empty directories being downloaded
"successfully". '*' is a special case for file urls only at this point
so remove any entries that accidentlly make it in through url mapping.
(Bitbake rev: 1369bec2404d942acc3618a8d005ec6868dcfd41)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When writing mirror specifications, the current regexp syntax can be awkward
and hard to get it to do what you want. For example, extracting the 'basename'
of a repository:
PREMIRRORS = "git://.*/([^/]+/)*([^/]*) git://somewhere.org/somedir/\\2;protocol=file"
can now become:
PREMIRRORS = "git://.*/.* git://somewhere.org/somedir/BASENAME;protocol=file"
which is much clearer. A MIRRORNAME substitution is also added which contains
an encoded form of both host and path. One of the problems with the existing
regexp syntax is you couldn't access HOST information from PATH and vice-versa
which is an issue this patch also addresses.
Tests for the new syntax are also added.
(Bitbake rev: c6b1acbad7b3d2698530eb8b5249adb4ab95da21)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we build the mirror urls, its possible an error will occur. If it
does, it should just mean we don't attempt this mirror url. The current
code actually aborts *all* the mirrors, not just the failed url.
This patch catches and logs the exception allowing things to continue.
(Bitbake rev: c35cbd1a1403865cf4f59ec88e1881669868103c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For mirror mapping, we never use regexps for the type component of
the url. Doing so causes various slightly bizarre behaviour such as
https:// urls being translated to files:// urls which we have no handler
for. This patch forces the type matches to be direct, not regexp based
and gives the expected bahvriour.
(From Poky rev: 695d8aca0999d2d61970b990e83c3132ba6f12cb)
(Bitbake rev: 604df1b25cf114e083f52917df2df64e01279c25)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using ud.localpath as a basename when it points at a directory causes
problems. The supports_checksum() method gives a good indication of whether
ud.localpath can be used in the way we need.
(From Poky rev: 933ec8a44634e33f92f6f76de3a34094c3d63aa6)
(Bitbake rev: dcd79ae20ab2c72c3312b2251c2b6dc4cabe988e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We only consider mirror tarballs when the source and target urls are of
differing types. We also should clear all url paramters when handling
mirror tarballs.
(From Poky rev: da140b8b0b3dda5429f9eee68829ef5247cdfe12)
(Bitbake rev: 7619dcad29a6c2405b15a8fbadfa11e81b399ae1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This means that parameters in the source expression are used as part of the
match. Parameters in the destination are used explicitly in the final
url.
(From Poky rev: c465cb0c5c927dd41d96ad6d6fa1566349574bb7)
(Bitbake rev: 3c468ac3ed6d045561afce19b85ae9dd18d87cea)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we only consider one pass through the mirror list. This doesn't
catch cases where for example you might want to setup a mirror of a mirror
and allow multiple redirection. There is no reason we can't support this
and the patch loops through the list recursively now.
As a safeguard, it will stop if any duplicate urls are found, hence
avoiding circular dependency looping.
(From Poky rev: 0ec0a4412865e54495c07beea1ced8355da58073)
(Bitbake rev: e585730e931e6abdb15ba8a3849c5fd22845b891)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With support for things like git:// -> git:// urls, we need to be
more explicity about the mirrortarball check since we need to fall
through to the following code in other cases.
(From Poky rev: 28e858cd6f7509468ef3e527a86820b9e06044db)
(Bitbake rev: a2459f5ca2f517964287f9a7c666a6856434e631)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are no functionality changes in this change
(From Poky rev: d222ebb7c75d74fde4fd04ea6feb27e10a862bae)
(Bitbake rev: db62e109cc36380ff8b8918628c9dea14ac9afbc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This assists with build reproducuility. It also avoids errors if cwd
happens not to exist when we call into the fetcher. That situation
would be unusual but I hit it with the unit tests.
(From Poky rev: 86517af9e066c2da1d580fa66b7c7f0340f3403e)
(Bitbake rev: b886c6c15a58643e06ca5ad7a3ff1f7766e4f48c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are actually two problems to fix
1) The exception for bb.process.ExecutionError must be processed first
because it is a derived from the bb.process.CmdError class and
we never reach the ExecutionError otherwise.
2) The stderr needs to be printed as well as stdout to determine
the root cause of a fetch failure.
The example I have is that I got a log that looked like:
--
ERROR: Function failed: Network access disabled through
BB_NO_NETWORK but access requested with command
/usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P
/localds 'http://downloads.yoctoproject.org/[...CLIPPED...] url None)
--
That really didn't tell me much, but with this patch I get error above
plus the following:
--
STDERR: /net/[...CLIPPED...]kernel-tools.git: Read-only file system
--
(Bitbake rev: af2133a04e1f4b22b181adf9c71f89c439bb88cf)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Skip evaluating remote URIs when doing local file checksums, because we
don't need to process them and doing so will trigger a parse failure if
SRCREV is not fully specified. Whilst this is just delaying a check
until runtime (when do_fetch runs for the recipe) we're only validating
this here accidentally and if we did wish to check it during parsing it
ought to be done explicitly.
Fixes [YOCTO #2512]
(Bitbake rev: 99feb77c2de707f2d825566cf48371c48f450e3e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This function no longer returns a value, it raises errors instead, so
update the comments to reflect this.
(Bitbake rev: 8b61b9584a8b00378f9fd8f165dbd65dcfd3c14e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Gathers a list of paths to have checksums calculated at parse time, and
processes these when calculating task hashes. Checksums are cached with
the file's current mtime. Thus, changing any local file in SRC_URI will
now cause the do_fetch taskhash to change, thus forcing a rebuild.
This change adds very roughly about an 8% increase in parse time (a few
seconds) and maybe a few seconds during runqueue generation, so a fairly
moderate performance hit.
Note that since paths are resolved at parse time, this will not force
a rebuild when files are introduced which would cause that resolved path
to be different - for example, where a machine-specific version of a file
was added without otherwise changing the recipe. This will need to be
handled in a future update.
Code to hook this into the signature generator was courtesy of
Richard Purdie <richard.purdie@linuxfoundation.org>.
Implements [YOCTO #2044].
(Bitbake rev: c993b7c457f8b7776e8a5dff253bfa0724bc2cae)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This ensures we can handle things like %-encoded characters in the path
portion of urls.
(Bitbake rev: b1dbc24ebcc4e5100c32568c2c41fd982fb4bcce)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this change, bitbake will try and checksum a wildcard which
will lead to fetch failures.
(Bitbake rev: ac53b88be58b0bed21730c0b61a8fc8e801a2f1b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In this way we avoid failing the build while trying to fetch local
directories.
[YOCTO #2475]
(Bitbake rev: 39adb5741d9eee0879d3181be505400dffc60804)
Signed-off-by: Andrei Gherzan <andrei@gherzan.ro>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There were some hardcoded behaviours in the system for which backends
support checksums verses which backends recommend them verses which
don't recommend them.
This moves the functionality into specific fetchers and then makes the
general code generic. This cleans up the codebase and fixes some corner
cases such as trying to checksum directories returned by the git fetcher.
(Bitbake rev: ef6d268f7b8527541a7fb044cf95a973be4097f4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, checksums are only checked for remote files. This changes
the check to apply to any file we have checksum data for. A mismatch
against a file is fatal but in the local case, no warnings are shown
about missing checksums.
(Bitbake rev: 80411b3acf2173a4a7d415102d16676eb98363e3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When fetching from mirrors, checksum errors would get buried in the
logs. This raises their profile so a warning is logged on the console
when fetcher checksum issues are encountered, even if other attempts
are made to get the file (which may or may not have the same issue).
(Bitbake rev: d43fafd7f01b5534499b45213197d8ccececdbc4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Under certain contitions such as local file:// mirrors of all files,
bitbake might not check checksums. This is not desirable and we should
always check the checksum where possible.
This only exception to this is the git mirror tarball case where the
checksum is unknown but we need to create the .done stamp for other
reasons. This patch preserves thta functionality but in a more specific
section of code where this doesn't interfere in normal checksums.
(Bitbake rev: 2592a43e422c06a6f6174ada562e9c8ba25e1ec1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton
Koen Kooi
Alejandro Hernandez
Richard Purdie
Roxana
Dennis Meier
Nick Lewis
Jacob Kroon
Olof Johansson
Robert Yang
Martin Jansa
Otavio Salvador
André Draszik
Peter Kjellerstedt
Matthew McClintock
Darren Hart
Tyler Hall
Constantin Musca
Christopher Larson
Mark Hatle
Andrei Gherzan
Jason Wessel
Christopher Larson