bitbake-selftest is failing due to directories not being created. This adds in an
appropriate mkdir so the tests can complete. Presumably in general OE use, something
else is ensuring the parent directory is created.
(Bitbake rev: 1270a07713e2a6c6e6fadcc61b785aebc99ae17b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Don't print the full exception in the initial warning - if we later
succeed in fetching the file from a mirror, we won't usually need the
details (which are in the fetch log if they are needed); otherwise the
full error will be printed when the fetch operation fails. Also adjust
the conditional block so that we don't print another warning just
mentioning we're going to try mirrors.
* Call logger.error() so that with knotty the full log is not printed
* Provide an explanation around the lines we print for easily updating
the checksums in the recipe. We don't want users to be just blindly
updating the recipe in case of a transient failure or deliberately
altered remote file.
(Bitbake rev: 2793413106c925b06783beb7413aa87cbcf246c3)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Most of the time we don't need to see the fetch command; the fetch log
includes the command as a debug message in any case, so omit it. Also
adjust the way command output is printed (we don't need stderr/stdout
labelled, and print "no output" instead of "output:\nNone" when there is
no output.
(Bitbake rev: a75505a52e4da918222100221f79e8a658f90446)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add in misssing space between the parameters. Reported by
Jate Sujjavanich <Jate.Sujjavanich@myfuelmaster.com>.
(Bitbake rev: 55382f0aac84b8f81cad0b82053c0b8295c33e54)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code in the CVS fetcher is elderly and there are simpler ways of
using the data store. This updates to use the modern APIs.
(Bitbake rev: 78eee8c70a80997293df99475153aed0b2ad0a17)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The localdata variable was removed, fix up a lost reference to this.
(Bitbake rev: 02ccc1396005ce0b7a2150a5ce12b723df21d464)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The implementation of NoMethodError and MalformedUrl was broken - if you
just set self.args in an exception class to a string it treats it as a
list and then fails later on with a TypeError due to the number of
arguments not matching up.
This nasty exception during exception handling was breaking the normal
exception flow (fixed separately), which meant that if you had a
malformed URL or invalid protocol in SRC_URI you would get the
following:
ERROR: Command execution failed: Traceback (most recent call last):
File "/home/user/poky/poky/bitbake/lib/bb/command.py", line 84, in runAsyncCommand
self.cooker.updateCache()
File "/home/user/poky/poky/bitbake/lib/bb/cooker.py", line 1207, in updateCache
if not self.parser.parse_next():
File "/home/user/poky/poky/bitbake/lib/bb/cooker.py", line 1694, in parse_next
logger.error('Unable to parse %s', value.recipe,
AttributeError: 'exceptions.TypeError' object has no attribute 'recipe'
A specific fix for [YOCTO #2977].
(Bitbake rev: 9d4150d99051d24ff218e8a43664ceaf524b19c7)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This ensures that if all a MIRRORS entry does is add a slash, this does
not result in a circular loop.
Fixes [YOCTO #3073].
(Bitbake rev: 57055d337a2c9997a6e5d5bdabaec396e3e128e9)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Unpack the ".rpm" binary package (only .src.rpm in the past)
* Unpack the .deb and .ipk binary package, their unpack commands are the same.
* This is useful for binary package recipe.
[YOCTO #1592]
(Bitbake rev: de7ceb9459574f33920ccc06255b533434f0ec25)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you have foo and foo.git in GITDIR, the two can end up being confused
by git with some horrible union of the two being cloned. This adds
a workaround to avoid this happening until git 1.7.9.2 onwards is
common enough for this to be removed. We use a symlink to hide
the directories we don't want git to know about.
(Bitbake rev: bbf1f6fe594c721a296ca09ee7c583d4a205c591)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When a fetch failure occurs for a local file, this patch ensures we print the
locations searched making it easier for the user to debug the problem.
(Bitbake rev: a461adbc5f09b41c771a7603370f6f2d1299ae8e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* in form which can be copied into a recipe without modification
* like oe-classic did since:
http://git.openembedded.org/openembedded/commit/?id=68abc465559a68e9201c9525be3c4acc6297eaed
* it shows them in right form when they are missing completely, but in
more verbose form when different
* it needs to print that only when checksums were requested, e.g.
fetching from sstate mirror sets both md5mismatch and sha256mismatch,
but your checksums shouldn't be shown
(Bitbake rev: 9eb34fc866775fd8310759a0111f232a9dc98981)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a file:// mirror is being used, the fetcher will create a symlink to the
local file. However, if the local file gets removed, that link will be dead,
and os.path.exists() returns False in that case, so it tries and fails to
recreate the link. Now we unlink such a dead link if it exists.
(Bitbake rev: 229ed3857e826e3e215e843cb51f729c1e13ed37)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without the new exception, when the system is configured to use premirrors,
but not allow network access (via BB_NO_NETWORK), when a recipe was lacking a
checksum the wrong error message(s) were being generated.
Instead of complaining about trying to perform network access, if the system
was able to find the item in the premirror, it should inform the user of the
lack of checksums, and the two SRC_URI fields they should use to update their
recipe.
(Bitbake rev: cb10e9c03a3f96d94e27e18330009616dde5e2b3)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This enhances the fetcher to allow preservation of the path component in urls
like: file://xxx/yyy/somefile.patch.
(Bitbake rev: e49a656a499355a5c6e7eb00bf5b8f1795e8dddb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
this allows wget fetcher to store the downloaded file in a specified custom filename in ${DL_DIR}
Exmaple:
SRC_URI = "https://edc.intel.com/Download.aspx?id=6190;downloadfilename=LIN_IEMGD_1_14_GOLD_2443.tgz"
This fixes bug:
[YOCTO #2570]
(Bitbake rev: ceb5871007f221c4d86a7bee421d4dd8d9100aaf)
Signed-off-by: Nitin A Kamble <nitin.a.kamble@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This problem occurs when fetching a different revision of the same source. Which mean every time you update a bzr package.
Using branch sets the pull location, and are the preferred way of cloning/branching a repository in bzr.
(Bitbake rev: 877a04d0b3cea9d5dbdf3c54fe0feb54cb997dda)
Signed-off-by: Martin Ertsaas <mertsas@cisco.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* it was send in v1 of proto -> protocol changes but then wasn't in V2
http://patchwork.openembedded.org/patch/31617/
where warning about proto= was moved to shared __init__
(Bitbake rev: b2017f493ab730d804ae44ec5a168d464626d046)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As well, if "proto" is used, get the associated value as "protocol"
(Bitbake rev: 53e6b630f0463d2d07cdaa9c9eb36794dc9b6b69)
Signed-off-by: Andrei Gherzan <andrei@gherzan.ro>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This also changes to use -t 2 -T 30 as the defaults which are more sane for
a modern fetcher and already specified in OpenEmbedded metadata.
(Bitbake rev: bf0e5dddf0f63cdb0648fb6d872af5ceef6fbfb0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This brings the cvs fetcher more into line with the others and allows consistent usage
of the FETCHCMD variable and option handling.
(Bitbake rev: 390ad59739356422852e976fa246375abc6aba08)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
svn changed working checkout formats between 1.6 and 1.7. Its convoluted to
detect what format a given working copy is in so the simplest solution is simply
to run "svn upgrade" within the working copy.
The base svn command variable is relocated slightly to enable this new code to
work effectively.
(Bitbake rev: ebd3ecdb5f3c6d3fe1fad27cbed4d80f4277e92e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
'git remote prune' at this location does not make much sense because
the following 'git remote rm' will prune stale and non-stale branches.
The 'prune' can cause trouble because it will access the network
bypassing the no-network code in bitbake. When this operation fails and
throws an exception, the next command (--> 'git remote rm') will be
skipped. This in turn, will make all the following operations fail,
because they assume that the remote does not exist yet.
(Bitbake rev: 2ba23df5fad4b94d38a6aed97f7822226d72eb89)
Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
People are using regexps in the url type field so we need to preserve
this bitbake behaviour. To address the issues with https:// urls mapping
badly to file:// urls we anchor the regexp if its not already anchored.
There should be no expressions in the wild which would break with this
change.
(Bitbake rev: ce0579dc256251e523c6330641f98b9f5a0e5761)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If '*' does end up in mirror urls accidently, some strange things
can break since supports_checksum() looks for this, ud.localpath can
then get ignored and this can lead to empty directories being downloaded
"successfully". '*' is a special case for file urls only at this point
so remove any entries that accidentlly make it in through url mapping.
(Bitbake rev: 1369bec2404d942acc3618a8d005ec6868dcfd41)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When writing mirror specifications, the current regexp syntax can be awkward
and hard to get it to do what you want. For example, extracting the 'basename'
of a repository:
PREMIRRORS = "git://.*/([^/]+/)*([^/]*) git://somewhere.org/somedir/\\2;protocol=file"
can now become:
PREMIRRORS = "git://.*/.* git://somewhere.org/somedir/BASENAME;protocol=file"
which is much clearer. A MIRRORNAME substitution is also added which contains
an encoded form of both host and path. One of the problems with the existing
regexp syntax is you couldn't access HOST information from PATH and vice-versa
which is an issue this patch also addresses.
Tests for the new syntax are also added.
(Bitbake rev: c6b1acbad7b3d2698530eb8b5249adb4ab95da21)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we build the mirror urls, its possible an error will occur. If it
does, it should just mean we don't attempt this mirror url. The current
code actually aborts *all* the mirrors, not just the failed url.
This patch catches and logs the exception allowing things to continue.
(Bitbake rev: c35cbd1a1403865cf4f59ec88e1881669868103c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If -l is specified and the source and destination are not on a common
filesystem an error occurs. The -l option is however the default for
git for local paths which the fetcher already now ensures in the
file:// case.
We can therefore safely drop the -l option.
(Bitbake rev: 3aeb268b2aaab4bb8b1cfff1450e0b76aa8ce855)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A file:// url should use "clone -l" to greatly speed
up the clone in the case of a kernel when it is local.
(Bitbake rev: 2bab2cc3ffe67ee2a308074a6e4c2c7be5636d2f)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For mirror mapping, we never use regexps for the type component of
the url. Doing so causes various slightly bizarre behaviour such as
https:// urls being translated to files:// urls which we have no handler
for. This patch forces the type matches to be direct, not regexp based
and gives the expected bahvriour.
(From Poky rev: 695d8aca0999d2d61970b990e83c3132ba6f12cb)
(Bitbake rev: 604df1b25cf114e083f52917df2df64e01279c25)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using ud.localpath as a basename when it points at a directory causes
problems. The supports_checksum() method gives a good indication of whether
ud.localpath can be used in the way we need.
(From Poky rev: 933ec8a44634e33f92f6f76de3a34094c3d63aa6)
(Bitbake rev: dcd79ae20ab2c72c3312b2251c2b6dc4cabe988e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We only consider mirror tarballs when the source and target urls are of
differing types. We also should clear all url paramters when handling
mirror tarballs.
(From Poky rev: da140b8b0b3dda5429f9eee68829ef5247cdfe12)
(Bitbake rev: 7619dcad29a6c2405b15a8fbadfa11e81b399ae1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This means that parameters in the source expression are used as part of the
match. Parameters in the destination are used explicitly in the final
url.
(From Poky rev: c465cb0c5c927dd41d96ad6d6fa1566349574bb7)
(Bitbake rev: 3c468ac3ed6d045561afce19b85ae9dd18d87cea)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we only consider one pass through the mirror list. This doesn't
catch cases where for example you might want to setup a mirror of a mirror
and allow multiple redirection. There is no reason we can't support this
and the patch loops through the list recursively now.
As a safeguard, it will stop if any duplicate urls are found, hence
avoiding circular dependency looping.
(From Poky rev: 0ec0a4412865e54495c07beea1ced8355da58073)
(Bitbake rev: e585730e931e6abdb15ba8a3849c5fd22845b891)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With support for things like git:// -> git:// urls, we need to be
more explicity about the mirrortarball check since we need to fall
through to the following code in other cases.
(From Poky rev: 28e858cd6f7509468ef3e527a86820b9e06044db)
(Bitbake rev: a2459f5ca2f517964287f9a7c666a6856434e631)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are no functionality changes in this change
(From Poky rev: d222ebb7c75d74fde4fd04ea6feb27e10a862bae)
(Bitbake rev: db62e109cc36380ff8b8918628c9dea14ac9afbc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This assists with build reproducuility. It also avoids errors if cwd
happens not to exist when we call into the fetcher. That situation
would be unusual but I hit it with the unit tests.
(From Poky rev: 86517af9e066c2da1d580fa66b7c7f0340f3403e)
(Bitbake rev: b886c6c15a58643e06ca5ad7a3ff1f7766e4f48c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The commit 420eb112a4f (fetch2: quote/unquote url paths) caused a regression
since local file urls containing quoted characters are no longer being handled
correctly. This fixes things so the url is dequoted correctly, fixing various
fetcher failures with recipes like "gtk+".
[YOCTO #2558]
(Bitbake rev: 95c0595d66b3f8a5f0716662ba2a878600f312ea)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are actually two problems to fix
1) The exception for bb.process.ExecutionError must be processed first
because it is a derived from the bb.process.CmdError class and
we never reach the ExecutionError otherwise.
2) The stderr needs to be printed as well as stdout to determine
the root cause of a fetch failure.
The example I have is that I got a log that looked like:
--
ERROR: Function failed: Network access disabled through
BB_NO_NETWORK but access requested with command
/usr/bin/env wget -t 5 -nv --passive-ftp --no-check-certificate -P
/localds 'http://downloads.yoctoproject.org/[...CLIPPED...] url None)
--
That really didn't tell me much, but with this patch I get error above
plus the following:
--
STDERR: /net/[...CLIPPED...]kernel-tools.git: Read-only file system
--
(Bitbake rev: af2133a04e1f4b22b181adf9c71f89c439bb88cf)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Skip evaluating remote URIs when doing local file checksums, because we
don't need to process them and doing so will trigger a parse failure if
SRCREV is not fully specified. Whilst this is just delaying a check
until runtime (when do_fetch runs for the recipe) we're only validating
this here accidentally and if we did wish to check it during parsing it
ought to be done explicitly.
Fixes [YOCTO #2512]
(Bitbake rev: 99feb77c2de707f2d825566cf48371c48f450e3e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This function no longer returns a value, it raises errors instead, so
update the comments to reflect this.
(Bitbake rev: 8b61b9584a8b00378f9fd8f165dbd65dcfd3c14e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Replace os.popen with subprocess.Popen since the older function would
fail (more or less) silently if the executed program cannot be found
There is a bb.process.run() which will invoke the Popen to run command,
use it for simplify the code.
For the:
p4file = os.popen("%s%s files %s" % (p4cmd, p4opt, depot))
...
for file in p4file:
list = file.split()
in bitbake/lib/bb/fetch2/perforce.py, it should be an error in the past,
since it didn't use readline() to read the pipe, but directly used the
split() for the pipe. Use the bb.process.run would fix the problem since
bb.process.run will return strings.
More info:
http://docs.python.org/library/subprocess.html#subprocess-replacements
[YOCTO #2075]
(Bitbake rev: 8d6700255a6d4dda403c89b171a6d4a1883e5aae)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Replace os.system with subprocess.call since the older function would
fail (more or less) silently if the executed program cannot be found
More info:
http://docs.python.org/library/subprocess.html#subprocess-replacements
[YOCTO #2075]
(Bitbake rev: f5b3bf115dc1ffbfb241a49cec0fc3654cb71021)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Gathers a list of paths to have checksums calculated at parse time, and
processes these when calculating task hashes. Checksums are cached with
the file's current mtime. Thus, changing any local file in SRC_URI will
now cause the do_fetch taskhash to change, thus forcing a rebuild.
This change adds very roughly about an 8% increase in parse time (a few
seconds) and maybe a few seconds during runqueue generation, so a fairly
moderate performance hit.
Note that since paths are resolved at parse time, this will not force
a rebuild when files are introduced which would cause that resolved path
to be different - for example, where a machine-specific version of a file
was added without otherwise changing the recipe. This will need to be
handled in a future update.
Code to hook this into the signature generator was courtesy of
Richard Purdie <richard.purdie@linuxfoundation.org>.
Implements [YOCTO #2044].
(Bitbake rev: c993b7c457f8b7776e8a5dff253bfa0724bc2cae)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This ensures we can handle things like %-encoded characters in the path
portion of urls.
(Bitbake rev: b1dbc24ebcc4e5100c32568c2c41fd982fb4bcce)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this change, bitbake will try and checksum a wildcard which
will lead to fetch failures.
(Bitbake rev: ac53b88be58b0bed21730c0b61a8fc8e801a2f1b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In this way we avoid failing the build while trying to fetch local
directories.
[YOCTO #2475]
(Bitbake rev: 39adb5741d9eee0879d3181be505400dffc60804)
Signed-off-by: Andrei Gherzan <andrei@gherzan.ro>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There were some hardcoded behaviours in the system for which backends
support checksums verses which backends recommend them verses which
don't recommend them.
This moves the functionality into specific fetchers and then makes the
general code generic. This cleans up the codebase and fixes some corner
cases such as trying to checksum directories returned by the git fetcher.
(Bitbake rev: ef6d268f7b8527541a7fb044cf95a973be4097f4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, checksums are only checked for remote files. This changes
the check to apply to any file we have checksum data for. A mismatch
against a file is fatal but in the local case, no warnings are shown
about missing checksums.
(Bitbake rev: 80411b3acf2173a4a7d415102d16676eb98363e3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When fetching from mirrors, checksum errors would get buried in the
logs. This raises their profile so a warning is logged on the console
when fetcher checksum issues are encountered, even if other attempts
are made to get the file (which may or may not have the same issue).
(Bitbake rev: d43fafd7f01b5534499b45213197d8ccececdbc4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Under certain contitions such as local file:// mirrors of all files,
bitbake might not check checksums. This is not desirable and we should
always check the checksum where possible.
This only exception to this is the git mirror tarball case where the
checksum is unknown but we need to create the .done stamp for other
reasons. This patch preserves thta functionality but in a more specific
section of code where this doesn't interfere in normal checksums.
(Bitbake rev: 2592a43e422c06a6f6174ada562e9c8ba25e1ec1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Under certain circumstances, bitbake could remove files which were
outside its control since it unconditionally removes ud.localpath.
In the file:// case this can point at external paths.
This patch converts it to use the clean() method which will remove
files when its safe to do so but not in the file:// case since the
file:// handler has an empty clean method. This means bitbake no
longer removes files outside its control and is generally much safer.
(Bitbake rev: 9ad1cebe2220b41da2141cdb3f0a403abb46d77c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With floating revisions and no specified branch, the fetcher could fail
with some obtuse errors. This was due to the branch name being set to None
which makes no sense. This patch reworks some conditions to avoid this.
(Bitbake rev: 740c58d43cfb1445dd126e4827bb70ce988ca107)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows some simplifications of the function and is slightly faster.
(Bitbake rev: 5999dc9985ad087c036611bfaa59b090a08781a6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: b766630ed7e099fffe817928a2811272677a1b26)
Signed-off-by: Jeff Polk <jeff.polk@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Don't check for network access before grabbing the the current head,
cloning, or updating a clone when the protocol is 'file'.
(Bitbake rev: d5847bc5254b9d2f28a6b574f6157d1286add27c)
Signed-off-by: Jeff Polk <jeff.polk@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As UI request, in recipes selection page, if user exclude a item,
the related depends recipes will be excluded together,so the view
clearly to add it.
[YOCTO #2100]
(Bitbake rev: c9eed04c6275ef2c694f89e047f85c7de76f89b6)
Signed-off-by: Liming An <limingx.l.an@intel.com>
Signed-off-by: Shane Wang <shane.wang@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 5a1e62a8cdd4f0253b0dc59046b3b4c3186af461)
Signed-off-by: Gary Thomas <gary@mlbassoc.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were seeing duplicate endings to urls:
DEBUG: For url ['http', 'www.apache.org', '/dist/subversion/subversion-1.7.1.tar.bz2', '', '', {}] comparing ['http', 'www.apache.org', '/dist', '', '', {}] to ['http', 'archive.apache.org', '/dist', '', '', {}]
DEBUG: For url http://www.apache.org/dist/subversion/subversion-1.7.1.tar.bz2 returning http://archive.apache.org/dist/subversion/subversion-1.7.1.tar.bz2/subversion-1.7.1.tar.bz2
This patch addresses this by only performing substitutions when really needed.
(Bitbake rev: a17473d51f54a9dfb4eeaa5ad516c4851d5ac142)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For similar reasons as the nocheckout option, packages that need
enhanced control over the checkout and branch creation on a repository
may want a complete mirror/bareclone created of the repository when
performing the unpack.
This is useful/required when a local respository is being used, but
local tracking branches have not been created for all branches that
a given recipe needs to manipulate. The standard git clone operations
will create remote branches for the branches that are local to the
source repository, but branches that are remote do not translate to
the destination repository. Doing a mirror/bare clone of the source,
makes all branches available to the repository.
This is a particular use case, but the ability to do a bare clone
creates great flexibility in recipe space, with no impact to recipes
that don't need this functionality.
To implement this, a new option 'bareclone' is craeted which creates
a mirror copy of the repository and leaves it bare in the unpacking
phase. A recipe that uses this option must both checkout and debare
the repository itself.
(Bitbake rev: 82482aae6f311c994275fb0b6b32d954bbfc78c3)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To quote my report of this to the git mailing list:
"""
I have a problem with git clone commands using alternates failing by
mixing up different repositories. I have a situation where I could end
up with both:
/srv/mirrors/repo
/srv/mirrors/repo.git
as bare clones.
I then try cloning "repo" with alternates with the command:
$ git clone -s -n /srv/mirrors/repo /tmp/foo
Cloning into /tmp/foo...
done.
$ cat /tmp/foo/.git/objects/info/alternates
/srv/mirrors/repo.git/objects
Note how I'm now referencing repo.git, not repo. This doesn't work as
expected giving some very bizarre results when actually using the
repository.
I appreciate this is a rather bizarre corner case but its one that is
breaking the build system I work with. Ideally people would use a
consistent URL for the same repository but we have an example where they
haven't and this really shouldn't break like this.
Looking at the code, the cause seems to be
clone.c:get_repo_path():
static char *suffix[] = { "/.git", ".git", "" };
since its looking in order for:
repo/.git (fails)
repo.git (suceeds, incorrect)
repo (never looked at)
I'm not sure what would break if that order were to change, swapping the
last two options.
I can "force" the issue by running:
git clone -s -n /srv/mirrors/repo/ /tmp/foo
but this results in the slightly odd looking:
$ cat /tmp/foo/.git/objects/info/alternates
/srv/mirrors/repo//objects
which does at least work.
"""
This patch adds the trailing slash to ensure the correct repository is
referenced at the expense of some ugliness in the alternates file.
(Bitbake rev: d978e7b35550e3785c7c567ffe4c40a3c3947450)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A lot of our checks in the fetcher code assume that if the downloaded
file exists, the download finished successfully; this meant that if
BitBake was interrupted in the middle of a fetch then it would not
resume the fetch the next time, but instead attempt to use the
half-fetched file and usually fail as a result.
Since we're already writing a ".done" stamp file when a fetch completes,
just check for its existence and assume the download didn't complete
successfully if it isn't present.
(Bitbake rev: 721e986624529aedef96dd06c9fe6243f2897b10)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make the warning for the initial fetch failure a single line - we don't
need the full command and output here yet, but write it into the log in
full as a debug message. However, if fetching from mirrors fails as well
then print out the full details for the first error that occurred as an
ERROR rather than a WARNING.
Since this is logged as an ERROR, combined with an earlier patch it
suppresses the full log which does make the output much more readable
for any fetch error.
Fixes [YOCTO #1832].
(Bitbake rev: 6bbdc7d259c0cc041b62dbdb26cfc3ec6edcb6f3)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The termination signal and exit code of the fetch process were not being
decoded correctly, resulting in bitbake reporting that the process
terminated with a signal of the exit code (if it was under 255). There
are functions in the Python os module to do this decoding correctly (for
Unix at least), so let's use them.
(Bitbake rev: 50aea9a76e40cf71cc3f1462c88298e4846a031c)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* The "name" argument to FuncFailed is rarely used as a name in actual
usage within bitbake, so don't treat it as one in the output.
* Don't print URL for FetchError if it was not specified (i.e. don't
output "Fetcher failure for URL 'None'")
* Don't include URL in "unable to fetch from any source" message since
we supply it to FetchError and it will be printed anyway.
* Don't include URL in "checksum failed" message for the same reason
(Bitbake rev: 86811bd85e2e453ee92a05fe60160d9b49ac69e8)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Local cloning of git repositories from DL_DIR into WORKDIR fails when
using ssh URL with port specification e.g.
"ssh://user@host:port/path/to/repo.git". Git fetcher clones such remote
repository into "${DL_DIR}/git2/host:port.path.to.repo.git". However,
when clonging from ${DL_DIR}/git2/host:port.path.to.repo.git into
${WORKDIR}, git fetcher fails with "ssh: Could not resolve hostname
${DLDIR}/git2/host: Name or service not known".
A solution is to replace ":" by "." in host component, similarly as it
is done when replacing "/" with "." in path component, so that local
clone directory in DL_DIR looks like this: "host.port.path.to.repo.git".
(Bitbake rev: 1f2867b79f1cd2bfbdc849ba5677a39db6fa3396)
Signed-off-by: Juraj Hercek <juraj.hercek@jhksoftware.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BB_STRICT_CHECKSUM caused a backtrace as the FetchError parameters are
incorrectly specified such that FetchError is being passed 8 params
when it's expecting 3. This fixes the parameters so we're passing a
formatted string and the url.
(Bitbake rev: b8b2f4287c9125542b18a294c0a94ed89a7e73a8)
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if you specify a mirror like:
file://.* http://linux.freescale.net/yocto/sstate-cache
it won't work as you expect whilst:
file://.* http://linux.freescale.net/yocto/sstate-cache/
will since it has the trailing slash.
This patch handles both cases correctly. It also adds some debug to
the uri_replace function since its near impossible to debug it without
some kind of output.
[YOCTO #1578]
(Bitbake rev: a0246bf09c93bb657eaf6ba61d090b247ed33640)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Defaulting to any file in DL_DIR as the first match for a file:// url
doesn't make much sense and can lead to unexpected results.
This patch changes the logic so this is the last fallback location
instead. Whether it should be using DL_DIR at all for this is a
good question but something for another patch.
[YOCTO #1710]
(Bitbake rev: 5597a68fac0954c682b67471722c2643e2415f99)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch improves the usability of checksums by only requiring one checksum
be defined.
Further, checksum verification will provide as much information as possible
at, rather than a bit at a time. No longer will you need to run fetch, see an
md5sum mismatch, fix it, run fetch, seen an sha256sum mismatch, fix it and
fetch again. If neither checksum is defined we now report both missing sums
at once - rather than one after the other.
Finally, if both sums are incorrect, we'll report both incorrect sums at the
same time.
Fixes part of [YOCTO #1399]
(Bitbake rev: 30ca41e596955fcbbe1ae6f4436ecf14cef96f8d)
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
git could need these environment variables when working behind
a proxy
(Bitbake rev: dca46cc2e1c75b6add2c4801e2994a4812745f5b)
Signed-off-by: Matthew McClintock <msm@freescale.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
git fetches can fail (or at least return failed) when trying to
fetch and prune rebased branches. This patch simply adds a -f
to the git fetch command so these failure are ignore
Generally, if some SHA was rebased away it's not coming back so
there is no point in not doing this force
(Bitbake rev: a7b75e4db52445b30ec0fc0053dcf454f5f7d2db)
Signed-off-by: Matthew McClintock <msm@freescale.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The _build_revision method in Hg class gets called with the wrong number
of arguments. This tiny patch adds a 5th argument to the method
declaration to prevent python from throwing an exception.
(Bitbake rev: 623e9c7f7a9cf12b8c81c26cc608990682a601dd)
Signed-off-by: Julian Pidancet <julian.pidancet@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make the git fetcher's subpath (path within the git repo to fetch)
option set the destsuffix (destination directory) option by default.
This reverts the behaviour of subpath to the same as when it was
introduced.
Based on a patch by Dmitry Eremin-Solenikov <dbaryshkov@gmail.com>
(Bitbake rev: 3e7f8afeacf7c8c8de3e87778a3907e33d4a06b3)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* in some cases there could be output like this
ERROR: ld.so: object 'libpseudo.so' from LD_PRELOAD cannot be preloaded: ignored.
before wc -l output and returned 'output.split()[0] != 0' is always True
(Bitbake rev: 3a54dcc09a12406ec6cf22b4b1a2cc4fc203822c)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* logging/logger typo was fixed in 38a598731b49c8a0ba0ede570adc33eb1e848235
but debug level is still missing
(Bitbake rev: 9de432fe2348cdbc93037bb49abb508d1fd38336)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Change the wget fetcher to not emit the commands output when calling checkstatus, this
matches the behaviour of the git fetchers checkstatus() method.
(Bitbake rev: 5896d8ff7bcf85b3bab20fc27c4c15e87f86b4d9)
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 639db8c766cada7180f9447f51303f9b30d7e817)
Signed-off-by: Holger Hans Peter Freyther <holger@moiji-mobile.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a user has a ~/.gitconfig file, git fetch --all will reference it. To avoid
this we should run git fetch with an explicit url telling it to fetch all
references (which includes tags).
I'm assured this means git won't reference the file, see the discussion on the
git mailing list (subject Overriding ~/.gitconfig using GIT_CONFIG).
[YOCTO #1134]
(Bitbake rev: 8540c09d4e509e3277940464c71174d8a0aca6ab)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie
Paul Eggleton
Robert Yang
Martin Jansa
Christopher Larson
Mark Hatle
Nitin A Kamble
Martin Ertsaas
Andrei Gherzan
Enrico Scholz
Jason Wessel
Christopher Larson
Jeff Polk
Liming An
Gary Thomas
Adrien Bustany
Bruce Ashfield
Juraj Hercek
Joshua Lock
Matthew McClintock
Julian Pidancet
Holger Hans Peter Freyther