Use git ls-remote to implement checkstatus command for the git fetcher.
(Bitbake rev: 0ed281feb6d244d3700da484f4e83394ae394f93)
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if checksum mismatch, fetcher will try mirror,
and if mirror fetching fail, it will show error "Unable
to fetch URL xxx from from any source" which actually
hide the real reason and make user confuse.
so showing warning message will make it more clear.
Fix [YOCTO #1256]
(Bitbake rev: b3e924297d670963714343d02f7898798fec84fd)
Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch is for [YOCTO #1085] fix.
If the upstream fails a checksum, retry from the MIRROR before giving up.
This will add more robust fetching if an upstream serves a bad file or webpage.
fetching of distcc prior to the move from samba -> googlecode is a good example
of this.
(Bitbake rev: b631e922257de52bf2247c01152d9856c870e7d0)
Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the git fetcher can malfunction when branches change in remote
repositories since whilst the update code updates the "origin" remote,
this isn't linked to the local heads.
By passing the --mirror option to 'git clone' and 'git remote add',
linkage between the local heads and remote heads is created with a 1:1
mapping, hence all the appropriate heads are then updated correctly.
This fixes some issues which have been seen with the Yocto autobuilder
mirrors.
(Bitbake rev: 3725602ec53df116dc108b3197a426b86ca43d5f)
Signed-off-by: Richard Purdie <richard.purdie@linux-foundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When an invalid 'protocol' parameter is used in a git SRC_URI,
the error reported was not helpful:
ERROR: Function 'Fetcher failure for URL: 'None'.
<environment dump>
fatal: Could not make temporary directory: No such file or directory
So instead check that ud.proto is set to something valid, and if not
raise a meaningful ParameterError which explains that the protocol
type is the source of the problem.
This fixes bug [YOCTO #1142]
(Bitbake rev: a2a29b72275ab03a263f4479a590b92111a0d6a8)
Signed-off-by: Scott Garman <scott.a.garman@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update exception handling syntax to use the modern style:
except ExcType as localvar
(Bitbake rev: dbf5f42b06bef81749b13aa99945cc1292a6676d)
Signed-off-by: Scott Garman <scott.a.garman@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The symptom of this problem is something like a cvs url which specifies
a username where the username is then passed through to something like
an http mirror.
This patch fixes things by ensuring empty entries are preserved in the
new URL.
(Bitbake rev: c1d978d7bd1ac8eb1e2d50029ab2384be9f72fb4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The git command string logged via check_network_access() does not match
the actual command executed in a few places. Ensure that it does.
(Bitbake rev: 10f3ca52dc274cd8b240987cfd7cd003aeda7ab1)
Signed-off-by: Darren Hart <dvhart@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Replace a call to print() with logging.debug() and flesh out the
message to clarify the state being reported.
(Bitbake rev: 9a28f7744e2f4224e7c097b8c4c1d49731b9a47e)
Signed-off-by: Darren Hart <dvhart@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
current git fetcher unpack method only checkout index and working tree,
but did not did not update the git branch in ref/heads, so user may not
get right info in ${S} by using git.
this patch enhance the unpack by using git checkout to fix this issue.
Fix bug [YOCTO #1089]
(Bitbake rev: c0eb89054aef4957966f98b44e7f3cce14fb337a)
Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
make the nocheckout option format to be: default is "0",
use nocheckou=1 to set this option
with this patch, the format will be consistant with other bitbake options
like rebaseable
(Bitbake rev: bd51659f5ee521cb8e6631d5f26792ab573e6b30)
Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some upstream git repo may rebase in the future, which means current
revision may disappear from the upstream repo after the rebase.
current git fetcher can not handle this case, because the git mirror
tar ball is per repo, and may also change in the rebase and lost the
current revision info.
To fix this issue, this patch
- add rebaseable tag in the SRC_URI
- for rebaseable repo, make git mirror tar ball per revision, in this
case, even upstream rebase, the git mirror still has the current
revision info.
- for rebaseable repo, generate mirror tar ball by default, since the
repo may change in the future.
(Bitbake rev: 92701d4c5372db48847c70da4ebd0736d79fd54b)
Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- use os.chmod, not os.fchmod, as the latter is missing under pypy
- rearrange our imports a bit
- don't die if sqlite3 is missing shared cache support
(Bitbake rev: f229824dc9c453adf6067500e2bf6761536e4f2f)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In current git fetcher, tag does not work due to commit http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=5920e85c561624e657c126df58f5c378a8950bbc. Tag is not in sha256 form, so it will be treated invalid, and silently replaced by latest revision.
To fix it, this patch treat tag name as branches name, thus it will be handled correctly later. Thanks Richard for reviewing and proposing the better approach.
Fix [YOCTO #972]
CC: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Yu Ke <ke.yu@intel.com>
The ordering constrains on the urldata_init functions are not straight
forward. To avoid further problems, create a helper function to setup
the source revisions which the init functions can all at the appropriate
point.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix a bug where ud.branches were being referenced before it was set by
the git fetcher when using AUTOREV. To do this some ordering needed
to be changed. This fixes errors like:
ERROR: Error parsing /recipes-kernel/linux/rt-tests_git.bb: Failure expanding variable
SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception
AttributeError: 'FetchData' object has no attribute 'branches'
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the current implementation, file:// urls as used by sstate don't access the
mirror code, breaking sstate mirror support. This change enables the usual
mirror handling. To do this, we remove the localfile special case, using the basename
paramemter instead. We also ensure the downloads directory is checked for files.
The drawback of this change is that file urls containing "*" globing require special
casing in the core.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* This patch fixes a cosmetic issue currently we get with master
WARNING: /home/kraj/work/bitbake/lib/bb/fetch2/__init__.py:733:
DeprecationWarning: Call to deprecated function bb.mkdirhier: Please use bb.utils.mkdirhier instead. bb.mkdirhier("%s/%s" % (rootdir, destdir))
(Bitbake rev: 36fe59ce314c295d239b76de34c8714def2c32d5)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This create a clean() method in each of the fetcher modules
and correctly cleans the .done stamp file and lock files
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Enable the fetcher to be able to unpack and SRPM. By default the system will
unpack the contents of the SRPM into the WORKDIR.
A new syntax "unpack=file" was developed for the SRC_URI, to allow for a
recipe to extract a specific file within an SRPM. An unpack operation will
then be executed on the extracted file.
In order to apply extracted patches (or unpack files not specified with
unpack), you must specify the path using WORKDIR, i.e.:
file://${WORKDIR}/mypatch.patch
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
See the problems in http://bugs.python.org/issue1692335, need to set self.args
correctly.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Drop some old md5 functions since we have improved functionality now which includes
sha256 checksum support. This stops each download being md5 checksumed twice.
Also change ".md5" stamp extentions to ".done" to better describe its use as a
download complete marker file and no longer write the md5 sum to the files.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We no longer need index/workdir support in the mirror tree and it causes all
kind of reference naming problems.Simplifying the code to remove this and use
just bare clones addresses this problem.
We increase the "version" number on the mirror tarballs to reflect the change
and ensure older mirror tarballs are not used as they would break.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since local mirror fetches are always symlinked from the download directory
directly, there is no need for this premirrors hack which doesn't cover
mirrors and also abuses the localpath variable with inconsistent results.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When files are fetched from a mirror source that happens to be local,
ensure links are created for the file since subsequent fetch calls
can then follow the links to find files.
Any other approach such as the existing manipulations of localpath
internally to the fetcher are prone to errors, races and other issues.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When processing a cvs SRC_URI to a file:// mirror, the user and host information
will break the mirror processing. This patch addresses it by only constructing
valid urls.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* SRC_URI format:
the SRC_URI are extended to allow multiple src rev:
name=<name1>,<name2>,...<name-n>
branch=<branch1>,<branch2>,...,<branch-n>
also SRCREV can be defined with
SRCREV_<name1> = xxxxx
SRCREV_<name2> = xxxxx
* FetchData extention
to support multiple src rev, several FetchData data are added:
- FetchData.names: list of name in SRC_URI, one name per srcrev. name is the index of revision and branch
- FetchData.revisions: dictionary of name->revision.
- FetchData.branches: dictionary of name->branch.
For example, linux-yocto recipes becomes:
SRC_URI = "git://git.pokylinux.org/linux-yocto-2.6.37;protocol=git;branch=${KBRANCH},meta;name=machine,meta"
FetchData.names = ['machine', 'meta']
FetchData.revisions = { 'machine':xxxxx, 'meta':xxxxxx }
FetchData.branches = { 'machine':${KBRANCH}, 'meta':'meta'}
* generic revision handling extension
the related revision handling code in fetch2.__init__.py are changed accordingly. the major change is add name parameter to indicate which src rev to handling. originally there is one src rev per FetchData, so FetchData parameter is enough. now since one FetchData has multiple src rev, it is necessary to use FetchData + name to specifiy src rev.
* git extension
git fetcher are also revised to take advantage of the multiple src rev in FetchData. especially the download() method are enhanced to fetch multiple src rev.
* other fetcher (svn, hg, ...) does not support multiple src rev. they just sync the API to add name, and then simply ignore the name. no actually functional change
Signed-off-by: Yu Ke <ke.yu@intel.com>
Since we're now always providing the git source control files it becomes
pointless to handle the tarballs of specific git revisions so drop this
part of the fetcher.
Signed-off-by: Yu Ke <ke.yu@intel.com>
The git download method clones the git repository to the local machine. The unpack process
can be optimised to be a local to local machine clone or a direct readtree operation to the
destination using git.will clone git repo to local, so git unpack can be simplified
to only checkouting the code to the work dir. For fullclone case, we also
need to manually copy all the ref info, which is needed by the later do_kernel_checkout().
Rather than use hardlinks, we reference the repository using alternatives since the
download directory may be on a different filesystem.
[Change to use -s by Richard Purdie]
Signed-off-by: Yu Ke <ke.yu@intel.com>
Sometime user want a purely local fetching, i.e. using local mirror without
any remote netowrk access. BB_NO_NETWORK option is introduced for this purpose
check_network_access() is the guard for BB_NO_NETWOKR option. it should be
put in any place that fetcher use network access
Signed-off-by: Yu Ke <ke.yu@intel.com>
the download is to fetch the source from URL, the build_mirror_data is
to create the mirror tar ball. the original go() method mix them together,
it is more clean to split them.
Signed-off-by: Yu Ke <ke.yu@intel.com>
there is case that we need to distingush bb.fetch and bb.fetch2,
and use different API for bb.fetch and bb.fetch2. so it is necessary
to add version info for distinguish purpose
Signed-off-by: Yu Ke <ke.yu@intel.com>
change the unpack to use the urldata and rootdir parameter
- urldata is the FetchData instance
- rootdir is the dir to put the extracted source. the original unpack
use current dir (os.getcwd) as destination dir, which is not flexible
and error-prone (error will occur if caller not chdir to dest dir)
Signed-off-by: Yu Ke <ke.yu@intel.com>
we should cache SRCREV whenever possible, the only exception is
when SREREV is auto rev. so change the logic to only set __BB_DONT_CACHE
at SRCREV = "${AUTOREV}" case
Signed-off-by: Yu Ke <ke.yu@intel.com>
Current fetcher has annoying "SRCREVINACTION" deadlock,
which occurs when SRCREV=${AUTOREV}=@bb.fetch.get_srcrev():
get_srcrev()->setup_localpath()->srcrev_internal_helper()
->evaluate SRCREV->get_srcrev()
current fetcher resolve the deadlock by introducing a
"SRCREVINACTION" condition check. Althoguh it works, it is
indeed not clean.
This patch use antoehr idea to break the deadlock: break
the dependency among SRCREV and get_srcrev(), i.e. assign
a specific keyword "AUTOINC" to AUTOREV. when Fetcher meet
this keyword, it will check and set the latest revision to
urldata.revision. get_srcrev later can use the urldata.revision
for value evaluation(SRCPV etc). In this case, SRCREV no longer
depends on get_srcrev, and there is not deadlock anymore.
Signed-off-by: Yu Ke <ke.yu@intel.com>
FetchData has some fetch method specific data, and only fetch method knows how
to initialize it. originally it is mostly initialized in Fetch.localpath().
But now there is requirement to call Fetch.latest_revision() before
Fetch.localpath(), thus require another earlier place for initialization. so
urldata_init is introduced for this purpose. it will be called in FetchData:__init__
and make all the Fetch functions useable after that.
Signed-off-by: Yu Ke <ke.yu@intel.com>
bb.fetch2 is copied from bb.fetch, and has many bb.fetch referrence.
Fix these referrence with bb.fetch2 referrence
Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Joshua Lock
Richard Purdie
Yu Ke
Scott Garman
Darren Hart
Chris Larson
Khem Raj
Mark Hatle
Saul Wold
Bernhard Reutner-Fischer