Skip evaluating remote URIs when doing local file checksums, because we
don't need to process them and doing so will trigger a parse failure if
SRCREV is not fully specified. Whilst this is just delaying a check
until runtime (when do_fetch runs for the recipe) we're only validating
this here accidentally and if we did wish to check it during parsing it
ought to be done explicitly.
Fixes [YOCTO #2512]
(Bitbake rev: 99feb77c2de707f2d825566cf48371c48f450e3e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This function no longer returns a value, it raises errors instead, so
update the comments to reflect this.
(Bitbake rev: 8b61b9584a8b00378f9fd8f165dbd65dcfd3c14e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Replace os.popen with subprocess.Popen since the older function would
fail (more or less) silently if the executed program cannot be found
There is a bb.process.run() which will invoke the Popen to run command,
use it for simplify the code.
For the:
p4file = os.popen("%s%s files %s" % (p4cmd, p4opt, depot))
...
for file in p4file:
list = file.split()
in bitbake/lib/bb/fetch2/perforce.py, it should be an error in the past,
since it didn't use readline() to read the pipe, but directly used the
split() for the pipe. Use the bb.process.run would fix the problem since
bb.process.run will return strings.
More info:
http://docs.python.org/library/subprocess.html#subprocess-replacements
[YOCTO #2075]
(Bitbake rev: 8d6700255a6d4dda403c89b171a6d4a1883e5aae)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Replace os.system with subprocess.call since the older function would
fail (more or less) silently if the executed program cannot be found
More info:
http://docs.python.org/library/subprocess.html#subprocess-replacements
[YOCTO #2075]
(Bitbake rev: f5b3bf115dc1ffbfb241a49cec0fc3654cb71021)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Gathers a list of paths to have checksums calculated at parse time, and
processes these when calculating task hashes. Checksums are cached with
the file's current mtime. Thus, changing any local file in SRC_URI will
now cause the do_fetch taskhash to change, thus forcing a rebuild.
This change adds very roughly about an 8% increase in parse time (a few
seconds) and maybe a few seconds during runqueue generation, so a fairly
moderate performance hit.
Note that since paths are resolved at parse time, this will not force
a rebuild when files are introduced which would cause that resolved path
to be different - for example, where a machine-specific version of a file
was added without otherwise changing the recipe. This will need to be
handled in a future update.
Code to hook this into the signature generator was courtesy of
Richard Purdie <richard.purdie@linuxfoundation.org>.
Implements [YOCTO #2044].
(Bitbake rev: c993b7c457f8b7776e8a5dff253bfa0724bc2cae)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This ensures we can handle things like %-encoded characters in the path
portion of urls.
(Bitbake rev: b1dbc24ebcc4e5100c32568c2c41fd982fb4bcce)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this change, bitbake will try and checksum a wildcard which
will lead to fetch failures.
(Bitbake rev: ac53b88be58b0bed21730c0b61a8fc8e801a2f1b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In this way we avoid failing the build while trying to fetch local
directories.
[YOCTO #2475]
(Bitbake rev: 39adb5741d9eee0879d3181be505400dffc60804)
Signed-off-by: Andrei Gherzan <andrei@gherzan.ro>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There were some hardcoded behaviours in the system for which backends
support checksums verses which backends recommend them verses which
don't recommend them.
This moves the functionality into specific fetchers and then makes the
general code generic. This cleans up the codebase and fixes some corner
cases such as trying to checksum directories returned by the git fetcher.
(Bitbake rev: ef6d268f7b8527541a7fb044cf95a973be4097f4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, checksums are only checked for remote files. This changes
the check to apply to any file we have checksum data for. A mismatch
against a file is fatal but in the local case, no warnings are shown
about missing checksums.
(Bitbake rev: 80411b3acf2173a4a7d415102d16676eb98363e3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When fetching from mirrors, checksum errors would get buried in the
logs. This raises their profile so a warning is logged on the console
when fetcher checksum issues are encountered, even if other attempts
are made to get the file (which may or may not have the same issue).
(Bitbake rev: d43fafd7f01b5534499b45213197d8ccececdbc4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Under certain contitions such as local file:// mirrors of all files,
bitbake might not check checksums. This is not desirable and we should
always check the checksum where possible.
This only exception to this is the git mirror tarball case where the
checksum is unknown but we need to create the .done stamp for other
reasons. This patch preserves thta functionality but in a more specific
section of code where this doesn't interfere in normal checksums.
(Bitbake rev: 2592a43e422c06a6f6174ada562e9c8ba25e1ec1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Under certain circumstances, bitbake could remove files which were
outside its control since it unconditionally removes ud.localpath.
In the file:// case this can point at external paths.
This patch converts it to use the clean() method which will remove
files when its safe to do so but not in the file:// case since the
file:// handler has an empty clean method. This means bitbake no
longer removes files outside its control and is generally much safer.
(Bitbake rev: 9ad1cebe2220b41da2141cdb3f0a403abb46d77c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With floating revisions and no specified branch, the fetcher could fail
with some obtuse errors. This was due to the branch name being set to None
which makes no sense. This patch reworks some conditions to avoid this.
(Bitbake rev: 740c58d43cfb1445dd126e4827bb70ce988ca107)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows some simplifications of the function and is slightly faster.
(Bitbake rev: 5999dc9985ad087c036611bfaa59b090a08781a6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: b766630ed7e099fffe817928a2811272677a1b26)
Signed-off-by: Jeff Polk <jeff.polk@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Don't check for network access before grabbing the the current head,
cloning, or updating a clone when the protocol is 'file'.
(Bitbake rev: d5847bc5254b9d2f28a6b574f6157d1286add27c)
Signed-off-by: Jeff Polk <jeff.polk@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As UI request, in recipes selection page, if user exclude a item,
the related depends recipes will be excluded together,so the view
clearly to add it.
[YOCTO #2100]
(Bitbake rev: c9eed04c6275ef2c694f89e047f85c7de76f89b6)
Signed-off-by: Liming An <limingx.l.an@intel.com>
Signed-off-by: Shane Wang <shane.wang@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 5a1e62a8cdd4f0253b0dc59046b3b4c3186af461)
Signed-off-by: Gary Thomas <gary@mlbassoc.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were seeing duplicate endings to urls:
DEBUG: For url ['http', 'www.apache.org', '/dist/subversion/subversion-1.7.1.tar.bz2', '', '', {}] comparing ['http', 'www.apache.org', '/dist', '', '', {}] to ['http', 'archive.apache.org', '/dist', '', '', {}]
DEBUG: For url http://www.apache.org/dist/subversion/subversion-1.7.1.tar.bz2 returning http://archive.apache.org/dist/subversion/subversion-1.7.1.tar.bz2/subversion-1.7.1.tar.bz2
This patch addresses this by only performing substitutions when really needed.
(Bitbake rev: a17473d51f54a9dfb4eeaa5ad516c4851d5ac142)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For similar reasons as the nocheckout option, packages that need
enhanced control over the checkout and branch creation on a repository
may want a complete mirror/bareclone created of the repository when
performing the unpack.
This is useful/required when a local respository is being used, but
local tracking branches have not been created for all branches that
a given recipe needs to manipulate. The standard git clone operations
will create remote branches for the branches that are local to the
source repository, but branches that are remote do not translate to
the destination repository. Doing a mirror/bare clone of the source,
makes all branches available to the repository.
This is a particular use case, but the ability to do a bare clone
creates great flexibility in recipe space, with no impact to recipes
that don't need this functionality.
To implement this, a new option 'bareclone' is craeted which creates
a mirror copy of the repository and leaves it bare in the unpacking
phase. A recipe that uses this option must both checkout and debare
the repository itself.
(Bitbake rev: 82482aae6f311c994275fb0b6b32d954bbfc78c3)
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To quote my report of this to the git mailing list:
"""
I have a problem with git clone commands using alternates failing by
mixing up different repositories. I have a situation where I could end
up with both:
/srv/mirrors/repo
/srv/mirrors/repo.git
as bare clones.
I then try cloning "repo" with alternates with the command:
$ git clone -s -n /srv/mirrors/repo /tmp/foo
Cloning into /tmp/foo...
done.
$ cat /tmp/foo/.git/objects/info/alternates
/srv/mirrors/repo.git/objects
Note how I'm now referencing repo.git, not repo. This doesn't work as
expected giving some very bizarre results when actually using the
repository.
I appreciate this is a rather bizarre corner case but its one that is
breaking the build system I work with. Ideally people would use a
consistent URL for the same repository but we have an example where they
haven't and this really shouldn't break like this.
Looking at the code, the cause seems to be
clone.c:get_repo_path():
static char *suffix[] = { "/.git", ".git", "" };
since its looking in order for:
repo/.git (fails)
repo.git (suceeds, incorrect)
repo (never looked at)
I'm not sure what would break if that order were to change, swapping the
last two options.
I can "force" the issue by running:
git clone -s -n /srv/mirrors/repo/ /tmp/foo
but this results in the slightly odd looking:
$ cat /tmp/foo/.git/objects/info/alternates
/srv/mirrors/repo//objects
which does at least work.
"""
This patch adds the trailing slash to ensure the correct repository is
referenced at the expense of some ugliness in the alternates file.
(Bitbake rev: d978e7b35550e3785c7c567ffe4c40a3c3947450)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A lot of our checks in the fetcher code assume that if the downloaded
file exists, the download finished successfully; this meant that if
BitBake was interrupted in the middle of a fetch then it would not
resume the fetch the next time, but instead attempt to use the
half-fetched file and usually fail as a result.
Since we're already writing a ".done" stamp file when a fetch completes,
just check for its existence and assume the download didn't complete
successfully if it isn't present.
(Bitbake rev: 721e986624529aedef96dd06c9fe6243f2897b10)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make the warning for the initial fetch failure a single line - we don't
need the full command and output here yet, but write it into the log in
full as a debug message. However, if fetching from mirrors fails as well
then print out the full details for the first error that occurred as an
ERROR rather than a WARNING.
Since this is logged as an ERROR, combined with an earlier patch it
suppresses the full log which does make the output much more readable
for any fetch error.
Fixes [YOCTO #1832].
(Bitbake rev: 6bbdc7d259c0cc041b62dbdb26cfc3ec6edcb6f3)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The termination signal and exit code of the fetch process were not being
decoded correctly, resulting in bitbake reporting that the process
terminated with a signal of the exit code (if it was under 255). There
are functions in the Python os module to do this decoding correctly (for
Unix at least), so let's use them.
(Bitbake rev: 50aea9a76e40cf71cc3f1462c88298e4846a031c)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* The "name" argument to FuncFailed is rarely used as a name in actual
usage within bitbake, so don't treat it as one in the output.
* Don't print URL for FetchError if it was not specified (i.e. don't
output "Fetcher failure for URL 'None'")
* Don't include URL in "unable to fetch from any source" message since
we supply it to FetchError and it will be printed anyway.
* Don't include URL in "checksum failed" message for the same reason
(Bitbake rev: 86811bd85e2e453ee92a05fe60160d9b49ac69e8)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Local cloning of git repositories from DL_DIR into WORKDIR fails when
using ssh URL with port specification e.g.
"ssh://user@host:port/path/to/repo.git". Git fetcher clones such remote
repository into "${DL_DIR}/git2/host:port.path.to.repo.git". However,
when clonging from ${DL_DIR}/git2/host:port.path.to.repo.git into
${WORKDIR}, git fetcher fails with "ssh: Could not resolve hostname
${DLDIR}/git2/host: Name or service not known".
A solution is to replace ":" by "." in host component, similarly as it
is done when replacing "/" with "." in path component, so that local
clone directory in DL_DIR looks like this: "host.port.path.to.repo.git".
(Bitbake rev: 1f2867b79f1cd2bfbdc849ba5677a39db6fa3396)
Signed-off-by: Juraj Hercek <juraj.hercek@jhksoftware.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BB_STRICT_CHECKSUM caused a backtrace as the FetchError parameters are
incorrectly specified such that FetchError is being passed 8 params
when it's expecting 3. This fixes the parameters so we're passing a
formatted string and the url.
(Bitbake rev: b8b2f4287c9125542b18a294c0a94ed89a7e73a8)
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if you specify a mirror like:
file://.* http://linux.freescale.net/yocto/sstate-cache
it won't work as you expect whilst:
file://.* http://linux.freescale.net/yocto/sstate-cache/
will since it has the trailing slash.
This patch handles both cases correctly. It also adds some debug to
the uri_replace function since its near impossible to debug it without
some kind of output.
[YOCTO #1578]
(Bitbake rev: a0246bf09c93bb657eaf6ba61d090b247ed33640)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Defaulting to any file in DL_DIR as the first match for a file:// url
doesn't make much sense and can lead to unexpected results.
This patch changes the logic so this is the last fallback location
instead. Whether it should be using DL_DIR at all for this is a
good question but something for another patch.
[YOCTO #1710]
(Bitbake rev: 5597a68fac0954c682b67471722c2643e2415f99)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch improves the usability of checksums by only requiring one checksum
be defined.
Further, checksum verification will provide as much information as possible
at, rather than a bit at a time. No longer will you need to run fetch, see an
md5sum mismatch, fix it, run fetch, seen an sha256sum mismatch, fix it and
fetch again. If neither checksum is defined we now report both missing sums
at once - rather than one after the other.
Finally, if both sums are incorrect, we'll report both incorrect sums at the
same time.
Fixes part of [YOCTO #1399]
(Bitbake rev: 30ca41e596955fcbbe1ae6f4436ecf14cef96f8d)
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
git could need these environment variables when working behind
a proxy
(Bitbake rev: dca46cc2e1c75b6add2c4801e2994a4812745f5b)
Signed-off-by: Matthew McClintock <msm@freescale.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
git fetches can fail (or at least return failed) when trying to
fetch and prune rebased branches. This patch simply adds a -f
to the git fetch command so these failure are ignore
Generally, if some SHA was rebased away it's not coming back so
there is no point in not doing this force
(Bitbake rev: a7b75e4db52445b30ec0fc0053dcf454f5f7d2db)
Signed-off-by: Matthew McClintock <msm@freescale.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The _build_revision method in Hg class gets called with the wrong number
of arguments. This tiny patch adds a 5th argument to the method
declaration to prevent python from throwing an exception.
(Bitbake rev: 623e9c7f7a9cf12b8c81c26cc608990682a601dd)
Signed-off-by: Julian Pidancet <julian.pidancet@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make the git fetcher's subpath (path within the git repo to fetch)
option set the destsuffix (destination directory) option by default.
This reverts the behaviour of subpath to the same as when it was
introduced.
Based on a patch by Dmitry Eremin-Solenikov <dbaryshkov@gmail.com>
(Bitbake rev: 3e7f8afeacf7c8c8de3e87778a3907e33d4a06b3)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* in some cases there could be output like this
ERROR: ld.so: object 'libpseudo.so' from LD_PRELOAD cannot be preloaded: ignored.
before wc -l output and returned 'output.split()[0] != 0' is always True
(Bitbake rev: 3a54dcc09a12406ec6cf22b4b1a2cc4fc203822c)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* logging/logger typo was fixed in 38a598731b49c8a0ba0ede570adc33eb1e848235
but debug level is still missing
(Bitbake rev: 9de432fe2348cdbc93037bb49abb508d1fd38336)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Change the wget fetcher to not emit the commands output when calling checkstatus, this
matches the behaviour of the git fetchers checkstatus() method.
(Bitbake rev: 5896d8ff7bcf85b3bab20fc27c4c15e87f86b4d9)
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 639db8c766cada7180f9447f51303f9b30d7e817)
Signed-off-by: Holger Hans Peter Freyther <holger@moiji-mobile.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a user has a ~/.gitconfig file, git fetch --all will reference it. To avoid
this we should run git fetch with an explicit url telling it to fetch all
references (which includes tags).
I'm assured this means git won't reference the file, see the discussion on the
git mailing list (subject Overriding ~/.gitconfig using GIT_CONFIG).
[YOCTO #1134]
(Bitbake rev: 8540c09d4e509e3277940464c71174d8a0aca6ab)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use git ls-remote to implement checkstatus command for the git fetcher.
(Bitbake rev: 0ed281feb6d244d3700da484f4e83394ae394f93)
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if checksum mismatch, fetcher will try mirror,
and if mirror fetching fail, it will show error "Unable
to fetch URL xxx from from any source" which actually
hide the real reason and make user confuse.
so showing warning message will make it more clear.
Fix [YOCTO #1256]
(Bitbake rev: b3e924297d670963714343d02f7898798fec84fd)
Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch is for [YOCTO #1085] fix.
If the upstream fails a checksum, retry from the MIRROR before giving up.
This will add more robust fetching if an upstream serves a bad file or webpage.
fetching of distcc prior to the move from samba -> googlecode is a good example
of this.
(Bitbake rev: b631e922257de52bf2247c01152d9856c870e7d0)
Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the git fetcher can malfunction when branches change in remote
repositories since whilst the update code updates the "origin" remote,
this isn't linked to the local heads.
By passing the --mirror option to 'git clone' and 'git remote add',
linkage between the local heads and remote heads is created with a 1:1
mapping, hence all the appropriate heads are then updated correctly.
This fixes some issues which have been seen with the Yocto autobuilder
mirrors.
(Bitbake rev: 3725602ec53df116dc108b3197a426b86ca43d5f)
Signed-off-by: Richard Purdie <richard.purdie@linux-foundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When an invalid 'protocol' parameter is used in a git SRC_URI,
the error reported was not helpful:
ERROR: Function 'Fetcher failure for URL: 'None'.
<environment dump>
fatal: Could not make temporary directory: No such file or directory
So instead check that ud.proto is set to something valid, and if not
raise a meaningful ParameterError which explains that the protocol
type is the source of the problem.
This fixes bug [YOCTO #1142]
(Bitbake rev: a2a29b72275ab03a263f4479a590b92111a0d6a8)
Signed-off-by: Scott Garman <scott.a.garman@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update exception handling syntax to use the modern style:
except ExcType as localvar
(Bitbake rev: dbf5f42b06bef81749b13aa99945cc1292a6676d)
Signed-off-by: Scott Garman <scott.a.garman@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The symptom of this problem is something like a cvs url which specifies
a username where the username is then passed through to something like
an http mirror.
This patch fixes things by ensuring empty entries are preserved in the
new URL.
(Bitbake rev: c1d978d7bd1ac8eb1e2d50029ab2384be9f72fb4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The git command string logged via check_network_access() does not match
the actual command executed in a few places. Ensure that it does.
(Bitbake rev: 10f3ca52dc274cd8b240987cfd7cd003aeda7ab1)
Signed-off-by: Darren Hart <dvhart@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Replace a call to print() with logging.debug() and flesh out the
message to clarify the state being reported.
(Bitbake rev: 9a28f7744e2f4224e7c097b8c4c1d49731b9a47e)
Signed-off-by: Darren Hart <dvhart@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
current git fetcher unpack method only checkout index and working tree,
but did not did not update the git branch in ref/heads, so user may not
get right info in ${S} by using git.
this patch enhance the unpack by using git checkout to fix this issue.
Fix bug [YOCTO #1089]
(Bitbake rev: c0eb89054aef4957966f98b44e7f3cce14fb337a)
Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
make the nocheckout option format to be: default is "0",
use nocheckou=1 to set this option
with this patch, the format will be consistant with other bitbake options
like rebaseable
(Bitbake rev: bd51659f5ee521cb8e6631d5f26792ab573e6b30)
Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some upstream git repo may rebase in the future, which means current
revision may disappear from the upstream repo after the rebase.
current git fetcher can not handle this case, because the git mirror
tar ball is per repo, and may also change in the rebase and lost the
current revision info.
To fix this issue, this patch
- add rebaseable tag in the SRC_URI
- for rebaseable repo, make git mirror tar ball per revision, in this
case, even upstream rebase, the git mirror still has the current
revision info.
- for rebaseable repo, generate mirror tar ball by default, since the
repo may change in the future.
(Bitbake rev: 92701d4c5372db48847c70da4ebd0736d79fd54b)
Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- use os.chmod, not os.fchmod, as the latter is missing under pypy
- rearrange our imports a bit
- don't die if sqlite3 is missing shared cache support
(Bitbake rev: f229824dc9c453adf6067500e2bf6761536e4f2f)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In current git fetcher, tag does not work due to commit http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=5920e85c561624e657c126df58f5c378a8950bbc. Tag is not in sha256 form, so it will be treated invalid, and silently replaced by latest revision.
To fix it, this patch treat tag name as branches name, thus it will be handled correctly later. Thanks Richard for reviewing and proposing the better approach.
Fix [YOCTO #972]
CC: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Yu Ke <ke.yu@intel.com>
The ordering constrains on the urldata_init functions are not straight
forward. To avoid further problems, create a helper function to setup
the source revisions which the init functions can all at the appropriate
point.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix a bug where ud.branches were being referenced before it was set by
the git fetcher when using AUTOREV. To do this some ordering needed
to be changed. This fixes errors like:
ERROR: Error parsing /recipes-kernel/linux/rt-tests_git.bb: Failure expanding variable
SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception
AttributeError: 'FetchData' object has no attribute 'branches'
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the current implementation, file:// urls as used by sstate don't access the
mirror code, breaking sstate mirror support. This change enables the usual
mirror handling. To do this, we remove the localfile special case, using the basename
paramemter instead. We also ensure the downloads directory is checked for files.
The drawback of this change is that file urls containing "*" globing require special
casing in the core.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* This patch fixes a cosmetic issue currently we get with master
WARNING: /home/kraj/work/bitbake/lib/bb/fetch2/__init__.py:733:
DeprecationWarning: Call to deprecated function bb.mkdirhier: Please use bb.utils.mkdirhier instead. bb.mkdirhier("%s/%s" % (rootdir, destdir))
(Bitbake rev: 36fe59ce314c295d239b76de34c8714def2c32d5)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This create a clean() method in each of the fetcher modules
and correctly cleans the .done stamp file and lock files
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Enable the fetcher to be able to unpack and SRPM. By default the system will
unpack the contents of the SRPM into the WORKDIR.
A new syntax "unpack=file" was developed for the SRC_URI, to allow for a
recipe to extract a specific file within an SRPM. An unpack operation will
then be executed on the extracted file.
In order to apply extracted patches (or unpack files not specified with
unpack), you must specify the path using WORKDIR, i.e.:
file://${WORKDIR}/mypatch.patch
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
See the problems in http://bugs.python.org/issue1692335, need to set self.args
correctly.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Drop some old md5 functions since we have improved functionality now which includes
sha256 checksum support. This stops each download being md5 checksumed twice.
Also change ".md5" stamp extentions to ".done" to better describe its use as a
download complete marker file and no longer write the md5 sum to the files.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We no longer need index/workdir support in the mirror tree and it causes all
kind of reference naming problems.Simplifying the code to remove this and use
just bare clones addresses this problem.
We increase the "version" number on the mirror tarballs to reflect the change
and ensure older mirror tarballs are not used as they would break.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since local mirror fetches are always symlinked from the download directory
directly, there is no need for this premirrors hack which doesn't cover
mirrors and also abuses the localpath variable with inconsistent results.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When files are fetched from a mirror source that happens to be local,
ensure links are created for the file since subsequent fetch calls
can then follow the links to find files.
Any other approach such as the existing manipulations of localpath
internally to the fetcher are prone to errors, races and other issues.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When processing a cvs SRC_URI to a file:// mirror, the user and host information
will break the mirror processing. This patch addresses it by only constructing
valid urls.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* SRC_URI format:
the SRC_URI are extended to allow multiple src rev:
name=<name1>,<name2>,...<name-n>
branch=<branch1>,<branch2>,...,<branch-n>
also SRCREV can be defined with
SRCREV_<name1> = xxxxx
SRCREV_<name2> = xxxxx
* FetchData extention
to support multiple src rev, several FetchData data are added:
- FetchData.names: list of name in SRC_URI, one name per srcrev. name is the index of revision and branch
- FetchData.revisions: dictionary of name->revision.
- FetchData.branches: dictionary of name->branch.
For example, linux-yocto recipes becomes:
SRC_URI = "git://git.pokylinux.org/linux-yocto-2.6.37;protocol=git;branch=${KBRANCH},meta;name=machine,meta"
FetchData.names = ['machine', 'meta']
FetchData.revisions = { 'machine':xxxxx, 'meta':xxxxxx }
FetchData.branches = { 'machine':${KBRANCH}, 'meta':'meta'}
* generic revision handling extension
the related revision handling code in fetch2.__init__.py are changed accordingly. the major change is add name parameter to indicate which src rev to handling. originally there is one src rev per FetchData, so FetchData parameter is enough. now since one FetchData has multiple src rev, it is necessary to use FetchData + name to specifiy src rev.
* git extension
git fetcher are also revised to take advantage of the multiple src rev in FetchData. especially the download() method are enhanced to fetch multiple src rev.
* other fetcher (svn, hg, ...) does not support multiple src rev. they just sync the API to add name, and then simply ignore the name. no actually functional change
Signed-off-by: Yu Ke <ke.yu@intel.com>
Since we're now always providing the git source control files it becomes
pointless to handle the tarballs of specific git revisions so drop this
part of the fetcher.
Signed-off-by: Yu Ke <ke.yu@intel.com>
The git download method clones the git repository to the local machine. The unpack process
can be optimised to be a local to local machine clone or a direct readtree operation to the
destination using git.will clone git repo to local, so git unpack can be simplified
to only checkouting the code to the work dir. For fullclone case, we also
need to manually copy all the ref info, which is needed by the later do_kernel_checkout().
Rather than use hardlinks, we reference the repository using alternatives since the
download directory may be on a different filesystem.
[Change to use -s by Richard Purdie]
Signed-off-by: Yu Ke <ke.yu@intel.com>
Sometime user want a purely local fetching, i.e. using local mirror without
any remote netowrk access. BB_NO_NETWORK option is introduced for this purpose
check_network_access() is the guard for BB_NO_NETWOKR option. it should be
put in any place that fetcher use network access
Signed-off-by: Yu Ke <ke.yu@intel.com>
the download is to fetch the source from URL, the build_mirror_data is
to create the mirror tar ball. the original go() method mix them together,
it is more clean to split them.
Signed-off-by: Yu Ke <ke.yu@intel.com>
there is case that we need to distingush bb.fetch and bb.fetch2,
and use different API for bb.fetch and bb.fetch2. so it is necessary
to add version info for distinguish purpose
Signed-off-by: Yu Ke <ke.yu@intel.com>
change the unpack to use the urldata and rootdir parameter
- urldata is the FetchData instance
- rootdir is the dir to put the extracted source. the original unpack
use current dir (os.getcwd) as destination dir, which is not flexible
and error-prone (error will occur if caller not chdir to dest dir)
Signed-off-by: Yu Ke <ke.yu@intel.com>
Paul Eggleton
Robert Yang
Christopher Larson
Richard Purdie
Andrei Gherzan
Jeff Polk
Liming An
Gary Thomas
Adrien Bustany
Bruce Ashfield
Juraj Hercek
Joshua Lock
Matthew McClintock
Julian Pidancet
Martin Jansa
Holger Hans Peter Freyther
Yu Ke
Scott Garman
Darren Hart
Chris Larson
Khem Raj
Mark Hatle
Saul Wold