The extra directory next to the recipe should only be created if there
are files to put into it; currently only the npm plugin does this. I
didn't notice the issue earlier because the test was actually able to
succeed under these circumstances if the recipe file came first in the
directory listing, which was a fault in my original oe-selftest test;
apparently on some YP autobuilder machines the order came out reversed.
With this change we can put the oe-selftest test that highlighted the
issue back to the way it was, with an extra check to reinforce that only
a single file should be created.
(From OE-Core rev: b8b778345eb0997c2cd952a1f61fdd2050b6b894)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
"npm shrinkwrap" creates a file that ensures that the exact same
versions get fetched the next time the recipe is built. lockdown is
similar but also includes sha1sums of the modules thus validating they
haven't changed between builds. These ensure that the build is
reproducible.
Fixes [YOCTO #9225].
(From OE-Core rev: 277377f13b2b771915eb853e336ca24b84523ed1)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow plugins to create additional files to go alongside the recipe. The
plugins don't know what the output filename is going to be, so they need
to put the files in a temporary location and add them to an "extrafiles"
dict within extravalues where the destination filename is the key and
the temporary path is the value.
devtool add was also extended to ensure these files get moved in and
preserved upon reset if they've been edited by the user.
(From OE-Core rev: 334b9451111b7e3efbb43b3a4eecebcab8ec6f0e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the user specifies an npm:// URL then the fetcher needs npm to be
available to run, so check if it's available early rather than failing
later.
(From OE-Core rev: a08d12ad867c292f7474731a0fe5e51e712446d6)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than rolling all of an npm module's dependencies into the same
package, split them into one module per package, setting the SUMMARY and
PKGV values from the package.json file for each package. Additionally,
mark each package with the appropriate license using the license
scanning we already do, falling back to the license stated in the
package.json file for the module if unknown. All of this is mostly in
aid of ensuring all modules and their licenses now show up in the
manifests for the image.
Additionally we set the main LICENSE value more concretely once we've
calculated the per-package licenses, since we have more information at
that point.
(From OE-Core rev: 8226805f83d21e7c1d2ba21969f3e8ee4b137496)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Matching license texts directly to md5sums only goes so far. Some
licenses make the copyright statement an intrinsic part of the license
statement (e.g. MIT) which of course varies between projects. Also,
people often seem to take standard license texts such as GPLv2 and
reformat them cosmetically - re-wrapping lines at a different width or
changing quoting styles are seemingly popular examples. In order to
match license files to their actual licenses more effectively, "crunch"
out these elements before comparing to an md5sum. (The existing plain
md5sum matching has been left in since it's a shortcut, and our list of
crunched md5sums isn't a complete replacement for it.)
As always, this code isn't providing any guarantees (legal or otherwise)
that it will always get the license correct - as indicated by the
accompanying comments the LICENSE values it writes out to the recipe are
indicative and you should verify them yourself by looking at the
documentation supplied from upstream for the software being built if you
have any concerns.
(From OE-Core rev: 553bb4ea5d51be5179e7d8c019740cf61ece76ea)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For example, this picks up a file named MIT-LICENSE.txt.
(From OE-Core rev: 103b4d26b340cbdf70bf43906e293f3497671fdc)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make the following improvements to mapping items specified in
AC_CHECK_PROG, AC_PATH_PROG and AX_WITH_PROG to recipes/classes:
* Produce a map of native recipe -> binary for all binaries currently in
STAGING_BINDIR_NATIVE and use this when mapping items
* Add some more entries to the class map
* Ignore autotools binaries since they are covered by the inherit of
autotools
* Ignore coreutils-native since that would almost always be a bogus
dependency
(From OE-Core rev: 5614c5ae6a004d4367eccc34dd3cc7ee61fb7e57)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow for whitespace in appropriate places, and ensure we match all
whitespace chars not just the space character.
(This fixes extracting dependencies from tmux's configure.ac, for
example.)
(From OE-Core rev: 63524ac8093b734aa4f29f4ea47bcc036f748314)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Clearly I didn't test this part of the code - lists don't have an "add"
method. Needless to say I have tested it now.
(From OE-Core rev: 063ed9058a14775f77e7875d4f6ef5719fa03f18)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add detection for npm modules and support for extracting the name and
version from package.json as is usually part of an npm module contents.
Note: this will likely only produce a buildable recipe if you use an
npm:// URL; simply pointing to a node.js source repository isn't going
to fetch the module's dependencies. It also doesn't set up the
shrinkwrap/lockdown automatically, so there is some room for improvement
later.
Implements [YOCTO #8690].
(From OE-Core rev: 41d0e4d75f13b53a6c1b6a8df9be4742be7534e0)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for detecting a Linux kernel source tree and generating a
basic kernel recipe using meta-skeleton's linux-yocto-custom recipe as a
base.
Implements [YOCTO #8981].
(From OE-Core rev: 39cab544b80ca4450106c9ede3180929ba24703c)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Detect kernel modules by looking for #include <linux/module.h>, and
handle the various styles of Makefile that appear to be used. I was able
to use this code to successfully build a number of external kernel
modules I found.
Implements [YOCTO #8982].
(From OE-Core rev: a85604f2eb2438b4caf0832c2ea15b5822f7e9a1)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Package names are actually case sensitive near as I can tell, so we
shouldn't be lowercasing them everywhere.
* Look for CMake packages in pkgdata and map those back to recipes,
so we aren't dependent on the hardcoded mappings (though those are
still preserved).
* Avoid duplicates in the unmapped package list
(From OE-Core rev: 2ddad52ccca07245eea43d9b844c6c7d4b667ca3)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a means of extending the dependency extraction for autotools and
cmake.
Note: in order to have this work, you need to have an __init__.py in the
lib/recipetool directory within your layer along with the module
implementing the handlers, and the __init__.py needs to contain:
# Enable other layers to have modules in the same named directory
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
(From OE-Core rev: 915dea9f89cd737e5ba167c384e8d314c5c23c49)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Looking at Chris Larson's code for starting the user's editor for
"recipetool newappend" it was slightly better than what I wrote for
"devtool edit-recipe" in that it checks VISUAL as well as EDITOR and
defaults to vi if neither are set, so break this out to its own function
and call it from both places. The broken out version passes shell=True
however in case it's a more complicated command rather than just a name
of an executable.
(From OE-Core rev: 184a256931e8cdc7bea97a905c4e67a435964de0)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you specify a local directory which happens to be a git repository
with an origin remote (and it is in fact remote), we can use that for
SRC_URI as implemented by OE-Core revision
b143d414846854dc8b3e0a47358daf5646eded38, however we also need to set S
if the recipe is going to be of any use fetching from that SRC_URI
later.
(From OE-Core rev: 8bbbd2d63f1bc752f9a30054a089dc2caf5fd84c)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When recipetool create is given a URL that starts with http(s):// and
contains /git/, such as the URLs at git.yoctoproject.org, it's fairly safe
to assume it's a git repository and not something that should be fetched
with wget, so rewrite the URL.
(From OE-Core rev: 3ca04757a670e8b6f78799cc0454d75691809ac4)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When auto-detecting the name for a recipe from the URL, strip off any
parameters (";name=value...") before parsing the URL, otherwise this
just ends up in the recipe name which is undesirable.
(From OE-Core rev: d3c46b5d0abd56bcadd4f2f1ef985f13d67f605b)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Both BitBucket and GitHub provide "release" downloads that unfortunately
don't include the actual project name. For recipetool this means we have
to have special handling for these URLs or else the current code assumes
the name of the project is something like "v0.9.1". Borrow and adapt
some code from autospec to do this, and put it in its own function for
clarity.
(From OE-Core rev: e8435fde7f82c39cc0efe1d4bf043b29f63898e9)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
CMake supports a find_library() directive to find named libraries, so
detect dependencies from this.
(From OE-Core rev: d0265bc67f797ee4b7760cf37335994133809abf)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I was a little bit hasty in OE-Core revision
c2cc5abe34169eae92067d97ce1e747e7c1413f5 - it turns out BitBake's
fetcher code is not consistent in whether it logs something useful or
not; when fetching from an http URL it does but with a git repository
it doesn't. In advance of any major reworking of fetch error handling in
BitBake, let's just print the text of the exception and then we know we
have shown something to the user.
Additionally, we were only catching FetchException here but there are
several other classes of exception that the fetcher can raise (e.g.
MalformedUrl); catch the parent BBFetchException class instead so we
avoid tracebacks for those other classes as well.
(From OE-Core rev: 578d3873a6415c9203c185c21cff472f7d2dab02)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the URL ends in a / then we want to strip that off the path we split
out of the URL before calling os.path.basename() on it.
(From OE-Core rev: 308189beda8a31541481d09e3d5e86187e843d8d)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you specify a local directory which happens to be a git repository
with an origin remote (and it is in fact remote), we can use that for
SRC_URI rather than leaving it blank in the recipe.
(From OE-Core rev: b143d414846854dc8b3e0a47358daf5646eded38)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sometimes you don't want to build an entire project, just a subdirectory
of it; add a --src-subdir option to make that easier. (We still look for
a single subdirectory in what gets unpacked, e.g. what you might find
within a tarball, so whatever you specify with this option is added onto
the end of that.)
(From OE-Core rev: 59682d78f95732e014f78f13e0a05f843860d9bb)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Quoting is optional in CMakeLists.txt and is occasionally used, so strip
out quotes if they are present.
(From OE-Core rev: 4ffe2e1ec9df05b92a2ad5746fb0ca6d218fd77e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I found these when I was looking at libftdi and they seem to be
generic enough to show up in at least a couple of other packages so I
figure I'll add them.
(From OE-Core rev: 9fa3ff44e05930d4dfa153db777077e747ecbf45)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for extracting dependencies from CMakeLists.txt. There's
still a bunch of things missing that are outside the scope of OE-Core
and we still lack a proper extension mechanism, but this is a good
start.
This also adds an oe-selftest test to exercise the new code a bit.
Implements [YOCTO #7635].
(From OE-Core rev: 77e73e6930381fdbd6e78d3913d6467572e16568)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We want to specify dependencies on virtual/* rather than whatever
library is selected in the current configuration.
(From OE-Core rev: e1ac0c45b27ded9962edaf34597f827d0b41ba82)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some refactoring to allow access to the library/header/pkg-config
mappings and the DEPENDS / unmapped dependency output code from other
classes than AutotoolsRecipeHandler.
(From OE-Core rev: 40c10d998b90dd59c6d36c28f8ba11ec598bfa0f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The regex for detecting git URLs was unanchored, leading to it matching
where it shouldn't have. An example of where this went wrong was
http://taglib.github.io/releases/taglib-1.9.1.tar.gz.
(From OE-Core rev: bacff751c88b680fbfb07843b18c59c8bc80a9ea)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Support a number of macros from autoconf-archive when reading
configure.ac to extract dependencies.
(From OE-Core rev: ee977a62c58ded361c2abd78654bd25637fe9ea1)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are a few different macros that can be used to pick up these
tools, add support for them all.
(From OE-Core rev: 7dfff4b7f05653aea230294ff1a7c023730deff9)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The presence of BOOST_REQUIRE or AX_BOOST.* indicates that boost is a
dependency.
(From OE-Core rev: 02570b1fc31c7f4e9643aea8365806089622c0e7)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* The regexes for PKG_CHECK_MODULES / AC_CHECK_LIB were a bit too strict
and thus we were skipping some macros.
* Add support for PKG_CHECK_EXISTS
* Avoid duplicates in warning on missing pkg-config dependencies
* Ignore dependency on musl (since this may come up if it's the selected
C library)
(From OE-Core rev: c58669fb0977f7f0cb79f252484d5c5ef0dfb7e4)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sometimes you need to build a variant of a recipe for the build
host as well as for the target (i.e. BBCLASSEXTEND = "native"); add a
--also-native command line option to "recipetool create" that enables
this and plumb it through from an identical option for "devtool add".
(We could conceivably do the same for nativesdk, but I felt it might be
confusing within the context of the extensible SDK, where nativesdk
isn't really relevant to the user.)
(From OE-Core rev: f3bea83db173cce921a9a30f04e88b7e3ed98854)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the recipe name ends with -native then we should inherit native;
likewise if it starts with nativesdk- then inherit nativesdk.
(Note that the recipe name must actually be specified by the user in
order to trigger this - we won't do it based on any name auto-detected
from e.g. the tarball name.)
Since we're doing this based on the name, "devtool add" will also gain
this functionality automatically.
(From OE-Core rev: a216fb1f5953327790dce3d2c1a9af616c0b410d)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As a matter of general convention we expect recipe names to be lower
case; in fact some of the packaging backends insist upon it. Since in
this part of the code we're auto-determining the name, we should convert
the name to lowercase if it's not already so that we're following
convention and avoiding any validation issues later on.
(From OE-Core rev: 64986decbc11afa0d1e11251f5f7adcba1860d19)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Having fetched the source and unpacked it to a temporary directory, we
then move part of it to the destination directory, or if the source is at
the top level we move the whole temporary directory, but in the latter
case we were later attempting to delete the temporary directory which no
longer existed. Clear out the variable so that doesn't happen.
(From OE-Core rev: 91714a52e91cddba5a16c73cf5765d1f47f7856c)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* tar and binutils we can assume are there
* libsocket is only relevant on BSD systems, so we can ignore it.
* Detect more things implying gettext/intltool is needed
* Detect glib-2.0 requirement.
(From OE-Core rev: 2c4c78a6a9970533f3352f1067b2263f45098493)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Often the filename (e.g. source tarball) contains the name and version
of the software it contains.
(This isn't intended to be exhaustive, just to catch the common case.)
(From OE-Core rev: 944eacfb849ee69b41e12c9de4f264406281ac6a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some build systems (notably autotools) support declaring the name and
version of the program being built; since we need those for the recipe
we can attempt to extract them. It's a little fuzzy as they are often
omitted or may not be appropriately formatted for our purposes, but it
does work on a reasonable number of software packages to be useful.
(From OE-Core rev: 3b3fd33190d89c09e62126eea0e45aa84fe5442e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sometimes we want to force one handler to run before another; if the two
handlers are in different plugins that's difficult without some kind of
priority number, so add one and sort by it.
(From OE-Core rev: 0219d4fb9cefcee635387b46fc1d215f82753d92)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the user specifies a URL that just returns a web page, then it's
probably incorrect (or broken); attempt to detect this and show an error
if it's the case.
(From OE-Core rev: 83b1245b2638eb5d314fe663d33cd52a776a34a7)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If SRC_URI happened not to be in the pre-generated lines then this code
would error out. This is unlikely to happen with the way the create code
is structured at the moment, but handle it just in case.
(From OE-Core rev: 95d33e90f2d5d9dd5ccc950856b8a939fefb831e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In my testing here it appears make -qn returns an error (exit code 2)
whereas make -n doesn't; I can't immediately tell why based on the
documentation. We don't actually care for it to be quiet since we're
capturing the output, so let's just leave -q off and have this work
properly as a result.
(From OE-Core rev: 30c4cd9efdac400d713dff645f23f2627277d75a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a fetch error occurs, the fetcher already prints a reasonable error -
we don't need the traceback as well, so catch that and exit if it
occurs.
(From OE-Core rev: c2cc5abe34169eae92067d97ce1e747e7c1413f5)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When you grab a URL for a github repository you'll almost certainly find
it in https://github.com/path/to/repository.git format; but bitbake's
fetcher can't handle that because it'll see https:// at the start and
assume it should use wget to fetch it. If the URL starts with http:// or
https:// and the path part ends with .git then assume it's a git
repository and adjust it accordingly.
(From OE-Core rev: bdbc4cf41d30eddb8a9ed882dedcc1670ce8fdd6)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This should have been register_commands rather than register_command;
I used register_commands in devtool so lets change this here to be
consistent with that. (Since this is extensible through layers though we
need to remain compatible with the old name, so fall back to that if the
new function name isn't there.)
(From OE-Core rev: 1047f6592ac81643cd847f104da766dc4a4c81ea)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Paul Eggleton
Christopher Larson