[YOCTO #5571] -- https://bugzilla.yoctoproject.org/show_bug.cgi?id=5571
The following workflow (whether accidentally or deliberately) would
previously not result in a checksum error, but would be helpful to do
so:
- Write a recipe with correct checksums
- Fetch the sources for this recipe using bitbake
- Change the checksums
Since the bitbake fetcher writes a done stamp after the initial download
and does not verify the checksums again (even if they are changed in the
recipe afterwards), the change of checksums is silently ignored.
Fix this without the overhead of computing the checksums from scratch on
every do_fetch by storing them in pickled format in the done stamp and
verifying that they still match those in the recipe.
(Bitbake rev: fbd4a0d422cf7f43db2f9eab2e47c41246a9031e)
Signed-off-by: Clemens Lang <clemens.lang@bmw-carit.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix sustition for rc, beta and alpha releses from -N to N weight.
(Bitbake rev: 63a9e60a6d80cfd2693ec1a6359785dc19f98e1f)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for scan every version directory using _check_latest_version
makes code more robust because sometimes upstream projects publish
new directories without files, causing don't find version.
To support this new behaviour remove _check_latest_dir and replace for
_check_latest_version_by_dir,
(Bitbake rev: 1a75b3707743c32eec9d2cf566fb6bbea9f73784)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add _check_latest_version_by_dir this function provides support
for scan every directory newer than current dir in order to get
latest_versionstring, example:
http://somedoamin.com/project/v2.1/http://somedoamin.com/project/v3.0/
Change return of _vercmp from True/False to -1/0/1 to provide test
when current directory is equal to newer directory this helps to
scan the same directory to get minor versions, example:
http://somedoamin.com/project/v2.1/project-v2.1.2.tgzhttp://somedoamin.com/project/v2.1/project-v2.1.6.tgz
(Bitbake rev: 5f7c5eb218a221165f59a0f4dd48d2d97f756193)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to reduce code duplication now compile package_regex in
_init_regexes instead of make this decision at _check_latest_version,
(Bitbake rev: e7284e3ad0e7dd91ed59dfbf8450ef62e89c7e54)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for get group only if exist in regex, this enables to use
this function in _check_latestversion regardless if the regex is generic
or specified by REGEX_URI.
(Bitbake rev: 1127af5b8c458929c4685b0326f86870ed09442e)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When built package regex for use in searching upstream version in sites,
the package name need to be escaped to avoid usage of special regex
char.
For example when search for gtk+, '+' need to be escaped.
(Bitbake rev: 1aa1de4b0c5bd34466e04844bbc371933736be59)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tweak the comments on get_srcrev() to better describe its function.
(Bitbake rev: b4d40f1ac7b32990c456cce261f99a5a157b5ae5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When urls ends with trailing slash os.path.basename return "" [1]
and built urldata.localpath only with DL_DIR, it causes that
donestamp is built as DL_DIR + '.done' and seems that ssh resource
was already download.
[YOCTO #6448]
[1] https://docs.python.org/2/library/os.path.html#os.path.basename
(Bitbake rev: 47992591349bab2c12741b937096e41085399087)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the code ignores lightweight tags which has caused some user
complaints. We can't put the right search list in place easily since
the results don't come back in a good order, head happens to sort
before tags.
In the end I refactored the function so we get the complete list of
remotes and then we can filter it ourselves in the order we chose,
including checking for light weight tags, preferring the proper ones.
Hopefully this resolves the issues people have been seeing.
[YOCTO #6881]
(Bitbake rev: 07ad307065bb15a48f0015b9e4a643201abdc283)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if you reference a file url, its checksum is included in the
task hash, however if you change to a different file at a different
location, perhaps taking advantage of the FILESPATH functionality, the
system will not reparse the file in question and change its checksum to
match the new file.
To correctly handle this, the system not only needs to know if the
existing file still exists or not, but also check the existance
of every file it would have looked at when computing the original file.
We already do this in the bitbake parsing code for class inclusion. This
change uses the same technique to log the file list we looked at and
if files in these locations exist when they previously did not, to
invalidate and reparse the file.
Since data stored in the cache is flattened text, we have to use a string
form of the data and split on the ":" character which is ugly, but is
an internal detail we can improve later if a better method is found.
The cache version changes to trigger a reparse since the previous
cache data is now incompatible.
[YOCTO #7019]
(Bitbake rev: 6c0706a28d72c591f1b75b6e3f3b645859387c7e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In some cases for cache purpoes we not only need to know which file
is going to be used but also which paths were considered. Add a
localpaths method which includes the history.
The core which() funciton already supports this, this just extends
the function to preserve the extra data we need. localpath becomes
just a special case of the case with history.
(Bitbake rev: ea5efeac5c1f7986666c979f789786f29fc1619a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Hardcoding a temp directory is bad practice and leads to races between
the tests. There is no longer any good reason for doing this, drop it
and ensure the files get cleaned up correctly.
(Bitbake rev: 10a47b1ec7470c9e8c4ffe0bb35cdf6d1bb2ee2e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to fix multithread usage of latest_versionstring moves package_custom_regex_comp
from class to method level because need to be defined by package.
Remove code for build url's with /download suffix because it's deprecated since you can
specify the download directory using package_regex.inc file.
(Bitbake rev: 231cae9f9b552ec6737795c098d1de426b5adcbc)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we're fetching outside of the context of a recipe, it's handy to be
able to disable checksum functionality so you don't get a meaningless
warning about the signatures being missing.
(Bitbake rev: 49dbcfbc56a206964acc5de761bba31be0283ba1)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some upstream sites put the name of the package in the body of href tags,
i.e. <a href="#43">somepackage-v1.4.10.tar.gz </a>.
(Bitbake rev: 4fa8542a8880d5749fbb3382706e011b012024d0)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Validate if package contain version string if not return the current
version cases for spectrum-fw and corpus recipes.
_check_latest_version return the latest version available don't
take into account the current version previous this only return
the upstream version if it greater than the current version.
(Bitbake rev: 91a7ac8c7f87f98e366585cf9720ec35b0790bae)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
package_custom_regex_comp is built with the current package name and
then used to search upstream version this reduces custom regex'es in
sites that have different packages in the same directory.
(Bitbake rev: a69d5d48c4bf9a3df3121cc4e69ba5d7e947ad8d)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Latest version string only try to find latest directory when REGEX_URI
isn't specified to avoid unnecessary processing and makes code easier
(Bitbake rev: afc33ec7cdb7d8ee3602a23fa973551ca5510ac4)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Being able to query whether updated versions of a url are available
is useful, not least for the package reporting system. Since such code
is closely linked to the url type and the url itself, the fetcher
makes a locical place to contain this code.
For wget based urls this means taking upstream directory listings
and searching those for later versions, returning those that are
found.
The patch also adds unittests for this function so that if
improvements are made, the original test urls can be used
to evaulate the those changes.
This is based on code from Irina Patru <irina.patru@intel.com>.
(Bitbake rev: a8272e22b7819e0e8afd8e291d276f5f28fc0007)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Being able to generate a version string representing the most recent git commit
given git is useful, not least for the package reporting system.
This adds in a latest_versionstring method to the git fetcher
which allows users to query the latest version using ls-remote
and filtering the responses.
The patch also adds unittests for this function so that if
improvements are made, the original test urls can be used
to evaulate the those changes.
This is based on code from Irina Patru <irina.patru@intel.com>.
(Bitbake rev: f71c8c0354e87fed80bc845db6728e6e18ce9c4d)
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This makes it possble to fetch Gerrit review references which are
normally stored under refs/changes.
Please disregard previous patch with the same topic.
(Bitbake rev: 268e9c0c6830e8e621c418f20c2ca12dc840e48b)
Signed-off-by: Fredrik Svensson <fredrik.svensson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
See http://www.nongnu.org/lzip/lzip.html for details on the compression
format.
(Bitbake rev: 9027b1273b5405c7269b013604ab417771b5eafe)
Signed-off-by: Koen Kooi <koen@dominion.thruhere.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Forcing strict to be a string, to avoid problems when performing comparisons
[YOCTO #6762]
(Bitbake rev: b8ed2098bdea2afd93ab4e3e1b834f3a31cb60de)
Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fixes urls of the form file://some/path/file;subdir=b. It also
adds in a couple of tests so we now tests these corner cases.
(Bitbake rev: 46306912a96444790efa9418d934dfdd36773ba1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Check if the 'subdir' parameter exists and assign it to 'destdir' so that
files are copied in ${WORKDIR}/destdir. This fixes urls that are of the form
file://a;subdir=b.
(Bitbake rev: 836a986b365eb9798563ec08d90b346596de7791)
Signed-off-by: Roxana Ciobanu <roxana.ciobanu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I could apply the label "release_1.0" to a super project that contains
many sub projects. If my recipes have SRC_URI's that use that label but
grab different sub-folders, than there's a bug where the cached localfile
(tar.gz) will not be unique and reused at the wrong times.
SRC_URI = "p4://perforce::1234@//depot/SuperProject/MiniProjectAAA/...;label=release_1.0 \
p4://perforce::1234@//depot/SuperProject/MiniProjectBBB/...;label=release_1.0"
(Bitbake rev: 3b5b1703b77490116dda895b29737cea51a3d6a0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We should only add user/password options if they're specified as in the
fetch case. Patch from Volker Vogelhuber <v.vogelhuber@digitalendoscopy.de>
(Bitbake rev: 303e6256947f4df4f283b75b7ccfdffa72864d67)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The clearcase fetcher is used to retrieve files from a ClearCase
(http://en.wikipedia.org/wiki/Rational_ClearCase) repository.
Usage in the recipe:
SRC_URI = "ccrc://cc.example.org/ccrc;vob=/example_vob;module=/example_module"
SRCREV = "EXAMPLE_CLEARCASE_TAG"
PV = "${@d.getVar("SRCREV").replace("/", "+")}"
The fetcher uses the rcleartool or cleartool remote client, depending on which one is available.
Supported SRC_URI options are:
- vob
(required) The name of the clearcase VOB (with prepending "/")
- module
The module in the selected VOB (with prepending "/")
The module and vob parameters are combined to create
the following load rule in the view config spec:
load <vob><module>
- proto
http or https
Related variables:
CCASE_CUSTOM_CONFIG_SPEC
Write a config spec to this variable in your recipe to use it instead
of the default config spec generated by this fetcher.
Please note that the SRCREV loses its functionality if you specify
this variable. SRCREV is still used to label the archive after a fetch,
but it doesn't define what's fetched.
User credentials:
When using cleartool:
The login of cleartool is handled by the system. No special steps needed.
When using rcleartool:
In order to use rcleartool with authenticated users an `rcleartool login` is
necessary before using the fetcher.
(Bitbake rev: 76cff0aaad4cf10b325bceadd8ea90b3d75edcc2)
Signed-off-by: Dennis Meier <meier.dennis@siemens.com>
Reviewed-by: Roger Meier <r.meier@siemens.com>
Reviewed-by: Christian Liechti <christian.liechti@siemens.com>
Reviewed-by: Henrique Mendonca <henrique.mendonca@siemens.com>
Reviewed-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There may be a need to set the user for the transport rather than the subversion
command itself. Add a parameter to the url to allow this.
[YOCTO #6475]
(Bitbake rev: 56c294dc30b6c2575b1cf904e26b8b8bef7677c2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Handle the following situations in a URL (e.g. in SRC_URI):
* Trailing semicolon in a URL - this is now ignored.
* Parameter specified with no value (no equals sign). This still
produces an error, but at least it is MalformedUrl with a proper
message rather than "ValueError: need more than 1 value to unpack".
(Bitbake rev: bfd13dfbc4c9f1dd8315002271791b1d9e274989)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A fix for the former patch when checking out a repository with
username and password using HG
(Bitbake rev: 0e7b594ccbceb3149f38776cea204807031ef69f)
Signed-off-by: Volker Vogelhuber <v.vogelhuber@digitalendoscopy.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Ensure there is default command available if FETCHCMD_p4 isn't set.
Also clean up the getVar references whilst here.
(Bitbake rev: d150226d11d5f041f78c8c3ce4abc5465dbc81d8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 42b7be1a6c5ff559f4a213eed04b370ca955460d)
Signed-off-by: Nick Lewis <nick.lewis@usa.g4s.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The initial version of this patch was reverted in
0a94e568152de550dedc8135a766beb18bf064ab
However, it was later agreed upon that it was reverted due to
a misunderstanding during code review; the patch is ok.
This version of the patch also removes an unneeded initialization of
the 'checksum' variable outside the scopes it is being used in.
(Bitbake rev: 9ee19a3ca2f8e11a91f0289ea3486310c61d40f2)
Signed-off-by: Jacob Kroon <jacob.kroon@mikrodidakt.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Following up a previous patch for mercurial fetcher, I just fixed a
problem when calling update on a repository with subrepositories enabled.
Attached a patch that fixes this problem
>From caae519a2bd64bf7c729bb26aff344827def47fb Mon Sep 17 00:00:00 2001
From: Volker Vogelhuber <v.vogelhuber@digitalendoscopy.de>
Date: Tue, 29 Apr 2014 15:29:47 +0200
Subject: [PATCH] bitbake: - fixed authentication issues in case of using sub
repositories
(Bitbake rev: dfa041c940caad12da50126a559afc8de089eeda)
Signed-off-by: Volker Vogelhuber <v.vogelhuber@digitalendoscopy.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit 6dfb33fd7204e09815bde7ba5bf21941a73fe965.
It causes an unindeded change in behaviour.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Cleanup the fix done in f9416e76e272ba3249abff099f6f3a47fe82e03e.
Instead of adding continue statements we can just move the last
statements into the final else-clause.
(Bitbake rev: 6dfb33fd7204e09815bde7ba5bf21941a73fe965)
Signed-off-by: Jacob Kroon <jacob.kroon@mikrodidakt.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Clemens Lang
Aníbal Limón
Ross Burton
Richard Purdie
Paul Eggleton
Fredrik Svensson
Koen Kooi
Alejandro Hernandez
Roxana
Chad Nelson
Andre McCurdy
Dennis Meier
Volker Vogelhuber
Chris Morgan
Nick Lewis
Jacob Kroon