generic-poky/scripts/oe-pkgdata-util

329 lines
11 KiB
Plaintext
Raw Normal View History

Rework installation of dev, dbg, doc, and locale packages Use a similar mechanism that was previously used to install locales at rootfs generation time to install other "complementary" packages (e.g. *-dev packages) - i.e. install all of the explicitly requested packages and their dependencies, then get a list of the packages that were installed, and use that list to install the complementary packages. This has been implemented by using a list of globs which should make it easier to extend in future. The previous locale package installation code assumed that the locale packages did not have any dependencies that were not already installed; now that we are installing non-locale packages this is no longer correct. In practice only the rpm backend actually made use of this assumption, so it needed to be changed to call into the existing package backend code to do the complementary package installation rather than calling rpm directly. This fixes the doc-pkgs IMAGE_FEATURES feature to work correctly, and also ensures that all dev/dbg packages get installed for dev-pkgs/dbg-pkgs respectively even if the dependency chains between those packages was not ensuring that already. The code has also been adapted to work correctly with the new SDK-from-image functionality. To that end, an SDKIMAGE_FEATURES variable has been added to allow specifying what extra image features should go into the SDK (extra, because by virtue of installing all of the packages in the image into the target part of the SDK, we already include all of IMAGE_FEATURES) with a default value of "dev-pkgs dbg-pkgs". Fixes [YOCTO #2614]. (From OE-Core rev: 72d1048a8381fa4a8c4c0d082047536727b4be47) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-07-09 13:15:08 +00:00
#!/usr/bin/env python
# OpenEmbedded pkgdata utility
#
# Written by: Paul Eggleton <paul.eggleton@linux.intel.com>
#
# Copyright 2012-2013 Intel Corporation
Rework installation of dev, dbg, doc, and locale packages Use a similar mechanism that was previously used to install locales at rootfs generation time to install other "complementary" packages (e.g. *-dev packages) - i.e. install all of the explicitly requested packages and their dependencies, then get a list of the packages that were installed, and use that list to install the complementary packages. This has been implemented by using a list of globs which should make it easier to extend in future. The previous locale package installation code assumed that the locale packages did not have any dependencies that were not already installed; now that we are installing non-locale packages this is no longer correct. In practice only the rpm backend actually made use of this assumption, so it needed to be changed to call into the existing package backend code to do the complementary package installation rather than calling rpm directly. This fixes the doc-pkgs IMAGE_FEATURES feature to work correctly, and also ensures that all dev/dbg packages get installed for dev-pkgs/dbg-pkgs respectively even if the dependency chains between those packages was not ensuring that already. The code has also been adapted to work correctly with the new SDK-from-image functionality. To that end, an SDKIMAGE_FEATURES variable has been added to allow specifying what extra image features should go into the SDK (extra, because by virtue of installing all of the packages in the image into the target part of the SDK, we already include all of IMAGE_FEATURES) with a default value of "dev-pkgs dbg-pkgs". Fixes [YOCTO #2614]. (From OE-Core rev: 72d1048a8381fa4a8c4c0d082047536727b4be47) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-07-09 13:15:08 +00:00
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License version 2 as
# published by the Free Software Foundation.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
#
import sys
import os
import os.path
import fnmatch
import re
import optparse
from collections import defaultdict
Rework installation of dev, dbg, doc, and locale packages Use a similar mechanism that was previously used to install locales at rootfs generation time to install other "complementary" packages (e.g. *-dev packages) - i.e. install all of the explicitly requested packages and their dependencies, then get a list of the packages that were installed, and use that list to install the complementary packages. This has been implemented by using a list of globs which should make it easier to extend in future. The previous locale package installation code assumed that the locale packages did not have any dependencies that were not already installed; now that we are installing non-locale packages this is no longer correct. In practice only the rpm backend actually made use of this assumption, so it needed to be changed to call into the existing package backend code to do the complementary package installation rather than calling rpm directly. This fixes the doc-pkgs IMAGE_FEATURES feature to work correctly, and also ensures that all dev/dbg packages get installed for dev-pkgs/dbg-pkgs respectively even if the dependency chains between those packages was not ensuring that already. The code has also been adapted to work correctly with the new SDK-from-image functionality. To that end, an SDKIMAGE_FEATURES variable has been added to allow specifying what extra image features should go into the SDK (extra, because by virtue of installing all of the packages in the image into the target part of the SDK, we already include all of IMAGE_FEATURES) with a default value of "dev-pkgs dbg-pkgs". Fixes [YOCTO #2614]. (From OE-Core rev: 72d1048a8381fa4a8c4c0d082047536727b4be47) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-07-09 13:15:08 +00:00
def glob(args, usage, debug=False):
bitbake.conf/package: Collapse PKGDATA_DIR into a single machine specific directory Currently we have a hierarchy of pkgdata directories and the code has to put together a search path and look through each in turn until it finds the data it needs. This has lead to a number of hardcoded paths and file globing which is unpredictable and undesirable. Worse, certain tricks that should be easy like a GL specific package architecture become problematic with the curretn search paths. With the modern sstate code, we can do better and construct a single pkgdata directory for each machine in just the same way as we do for the sysroot. This is already tried and well tested. With such a single directory, all the code that iterated through multiple pkgdata directories and simply be removed and give a significant simplification of the code. Even existing build directories adapt to the change well since the package contents doesn't change, just the location they're installed to and the stamp for them. The only complication is the we need a different shlibs directory for each multilib. These are only used by package.bbclass and the simple fix is to add MLPREFIX to the shlib directory name. This means the multilib packages will repackage and the sstate checksum will change but an existing build directory will adapt to the changes safely. It is close to release however I believe the benefits this patch give us are worth consideration for inclusion and give us more options for dealing with problems like the GL one. It also sets the ground work well for shlibs improvements in 1.6. (From OE-Core rev: 1b8e4abd2d9c0901d38d89d0f944fe1ffd019379) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-09-13 12:35:31 +00:00
if len(args) < 3:
Rework installation of dev, dbg, doc, and locale packages Use a similar mechanism that was previously used to install locales at rootfs generation time to install other "complementary" packages (e.g. *-dev packages) - i.e. install all of the explicitly requested packages and their dependencies, then get a list of the packages that were installed, and use that list to install the complementary packages. This has been implemented by using a list of globs which should make it easier to extend in future. The previous locale package installation code assumed that the locale packages did not have any dependencies that were not already installed; now that we are installing non-locale packages this is no longer correct. In practice only the rpm backend actually made use of this assumption, so it needed to be changed to call into the existing package backend code to do the complementary package installation rather than calling rpm directly. This fixes the doc-pkgs IMAGE_FEATURES feature to work correctly, and also ensures that all dev/dbg packages get installed for dev-pkgs/dbg-pkgs respectively even if the dependency chains between those packages was not ensuring that already. The code has also been adapted to work correctly with the new SDK-from-image functionality. To that end, an SDKIMAGE_FEATURES variable has been added to allow specifying what extra image features should go into the SDK (extra, because by virtue of installing all of the packages in the image into the target part of the SDK, we already include all of IMAGE_FEATURES) with a default value of "dev-pkgs dbg-pkgs". Fixes [YOCTO #2614]. (From OE-Core rev: 72d1048a8381fa4a8c4c0d082047536727b4be47) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-07-09 13:15:08 +00:00
usage()
sys.exit(1)
pkgdata_dir = args[0]
bitbake.conf/package: Collapse PKGDATA_DIR into a single machine specific directory Currently we have a hierarchy of pkgdata directories and the code has to put together a search path and look through each in turn until it finds the data it needs. This has lead to a number of hardcoded paths and file globing which is unpredictable and undesirable. Worse, certain tricks that should be easy like a GL specific package architecture become problematic with the curretn search paths. With the modern sstate code, we can do better and construct a single pkgdata directory for each machine in just the same way as we do for the sysroot. This is already tried and well tested. With such a single directory, all the code that iterated through multiple pkgdata directories and simply be removed and give a significant simplification of the code. Even existing build directories adapt to the change well since the package contents doesn't change, just the location they're installed to and the stamp for them. The only complication is the we need a different shlibs directory for each multilib. These are only used by package.bbclass and the simple fix is to add MLPREFIX to the shlib directory name. This means the multilib packages will repackage and the sstate checksum will change but an existing build directory will adapt to the changes safely. It is close to release however I believe the benefits this patch give us are worth consideration for inclusion and give us more options for dealing with problems like the GL one. It also sets the ground work well for shlibs improvements in 1.6. (From OE-Core rev: 1b8e4abd2d9c0901d38d89d0f944fe1ffd019379) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-09-13 12:35:31 +00:00
pkglist_file = args[1]
globs = args[2].split()
Rework installation of dev, dbg, doc, and locale packages Use a similar mechanism that was previously used to install locales at rootfs generation time to install other "complementary" packages (e.g. *-dev packages) - i.e. install all of the explicitly requested packages and their dependencies, then get a list of the packages that were installed, and use that list to install the complementary packages. This has been implemented by using a list of globs which should make it easier to extend in future. The previous locale package installation code assumed that the locale packages did not have any dependencies that were not already installed; now that we are installing non-locale packages this is no longer correct. In practice only the rpm backend actually made use of this assumption, so it needed to be changed to call into the existing package backend code to do the complementary package installation rather than calling rpm directly. This fixes the doc-pkgs IMAGE_FEATURES feature to work correctly, and also ensures that all dev/dbg packages get installed for dev-pkgs/dbg-pkgs respectively even if the dependency chains between those packages was not ensuring that already. The code has also been adapted to work correctly with the new SDK-from-image functionality. To that end, an SDKIMAGE_FEATURES variable has been added to allow specifying what extra image features should go into the SDK (extra, because by virtue of installing all of the packages in the image into the target part of the SDK, we already include all of IMAGE_FEATURES) with a default value of "dev-pkgs dbg-pkgs". Fixes [YOCTO #2614]. (From OE-Core rev: 72d1048a8381fa4a8c4c0d082047536727b4be47) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-07-09 13:15:08 +00:00
if not os.path.exists(pkgdata_dir):
print('ERROR: Unable to find pkgdata directory %s' % pkgdata_dir)
sys.exit(1)
if not os.path.exists(pkglist_file):
print('ERROR: Unable to find package list file %s' % pkglist_file)
sys.exit(1)
Rework installation of dev, dbg, doc, and locale packages Use a similar mechanism that was previously used to install locales at rootfs generation time to install other "complementary" packages (e.g. *-dev packages) - i.e. install all of the explicitly requested packages and their dependencies, then get a list of the packages that were installed, and use that list to install the complementary packages. This has been implemented by using a list of globs which should make it easier to extend in future. The previous locale package installation code assumed that the locale packages did not have any dependencies that were not already installed; now that we are installing non-locale packages this is no longer correct. In practice only the rpm backend actually made use of this assumption, so it needed to be changed to call into the existing package backend code to do the complementary package installation rather than calling rpm directly. This fixes the doc-pkgs IMAGE_FEATURES feature to work correctly, and also ensures that all dev/dbg packages get installed for dev-pkgs/dbg-pkgs respectively even if the dependency chains between those packages was not ensuring that already. The code has also been adapted to work correctly with the new SDK-from-image functionality. To that end, an SDKIMAGE_FEATURES variable has been added to allow specifying what extra image features should go into the SDK (extra, because by virtue of installing all of the packages in the image into the target part of the SDK, we already include all of IMAGE_FEATURES) with a default value of "dev-pkgs dbg-pkgs". Fixes [YOCTO #2614]. (From OE-Core rev: 72d1048a8381fa4a8c4c0d082047536727b4be47) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-07-09 13:15:08 +00:00
skipregex = re.compile("-locale-|^locale-base-|-dev$|-doc$|-dbg$|-staticdev$|^kernel-module-")
mappedpkgs = set()
with open(pkglist_file, 'r') as f:
for line in f:
fields = line.rstrip().split()
if not fields:
Rework installation of dev, dbg, doc, and locale packages Use a similar mechanism that was previously used to install locales at rootfs generation time to install other "complementary" packages (e.g. *-dev packages) - i.e. install all of the explicitly requested packages and their dependencies, then get a list of the packages that were installed, and use that list to install the complementary packages. This has been implemented by using a list of globs which should make it easier to extend in future. The previous locale package installation code assumed that the locale packages did not have any dependencies that were not already installed; now that we are installing non-locale packages this is no longer correct. In practice only the rpm backend actually made use of this assumption, so it needed to be changed to call into the existing package backend code to do the complementary package installation rather than calling rpm directly. This fixes the doc-pkgs IMAGE_FEATURES feature to work correctly, and also ensures that all dev/dbg packages get installed for dev-pkgs/dbg-pkgs respectively even if the dependency chains between those packages was not ensuring that already. The code has also been adapted to work correctly with the new SDK-from-image functionality. To that end, an SDKIMAGE_FEATURES variable has been added to allow specifying what extra image features should go into the SDK (extra, because by virtue of installing all of the packages in the image into the target part of the SDK, we already include all of IMAGE_FEATURES) with a default value of "dev-pkgs dbg-pkgs". Fixes [YOCTO #2614]. (From OE-Core rev: 72d1048a8381fa4a8c4c0d082047536727b4be47) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-07-09 13:15:08 +00:00
continue
pkg = fields[0]
# We don't care about other args (used to need the package architecture but the
# new pkgdata structure avoids the need for that)
Rework installation of dev, dbg, doc, and locale packages Use a similar mechanism that was previously used to install locales at rootfs generation time to install other "complementary" packages (e.g. *-dev packages) - i.e. install all of the explicitly requested packages and their dependencies, then get a list of the packages that were installed, and use that list to install the complementary packages. This has been implemented by using a list of globs which should make it easier to extend in future. The previous locale package installation code assumed that the locale packages did not have any dependencies that were not already installed; now that we are installing non-locale packages this is no longer correct. In practice only the rpm backend actually made use of this assumption, so it needed to be changed to call into the existing package backend code to do the complementary package installation rather than calling rpm directly. This fixes the doc-pkgs IMAGE_FEATURES feature to work correctly, and also ensures that all dev/dbg packages get installed for dev-pkgs/dbg-pkgs respectively even if the dependency chains between those packages was not ensuring that already. The code has also been adapted to work correctly with the new SDK-from-image functionality. To that end, an SDKIMAGE_FEATURES variable has been added to allow specifying what extra image features should go into the SDK (extra, because by virtue of installing all of the packages in the image into the target part of the SDK, we already include all of IMAGE_FEATURES) with a default value of "dev-pkgs dbg-pkgs". Fixes [YOCTO #2614]. (From OE-Core rev: 72d1048a8381fa4a8c4c0d082047536727b4be47) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-07-09 13:15:08 +00:00
# Skip packages for which there is no point applying globs
if skipregex.search(pkg):
if debug:
print("%s -> !!" % pkg)
continue
# Skip packages that already match the globs, so if e.g. a dev package
# is already installed and thus in the list, we don't process it any further
# Most of these will be caught by skipregex already, but just in case...
already = False
for g in globs:
if fnmatch.fnmatchcase(pkg, g):
already = True
break
if already:
if debug:
print("%s -> !" % pkg)
continue
# Define some functions
def revpkgdata(pkgn):
bitbake.conf/package: Collapse PKGDATA_DIR into a single machine specific directory Currently we have a hierarchy of pkgdata directories and the code has to put together a search path and look through each in turn until it finds the data it needs. This has lead to a number of hardcoded paths and file globing which is unpredictable and undesirable. Worse, certain tricks that should be easy like a GL specific package architecture become problematic with the curretn search paths. With the modern sstate code, we can do better and construct a single pkgdata directory for each machine in just the same way as we do for the sysroot. This is already tried and well tested. With such a single directory, all the code that iterated through multiple pkgdata directories and simply be removed and give a significant simplification of the code. Even existing build directories adapt to the change well since the package contents doesn't change, just the location they're installed to and the stamp for them. The only complication is the we need a different shlibs directory for each multilib. These are only used by package.bbclass and the simple fix is to add MLPREFIX to the shlib directory name. This means the multilib packages will repackage and the sstate checksum will change but an existing build directory will adapt to the changes safely. It is close to release however I believe the benefits this patch give us are worth consideration for inclusion and give us more options for dealing with problems like the GL one. It also sets the ground work well for shlibs improvements in 1.6. (From OE-Core rev: 1b8e4abd2d9c0901d38d89d0f944fe1ffd019379) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-09-13 12:35:31 +00:00
return os.path.join(pkgdata_dir, "runtime-reverse", pkgn)
Rework installation of dev, dbg, doc, and locale packages Use a similar mechanism that was previously used to install locales at rootfs generation time to install other "complementary" packages (e.g. *-dev packages) - i.e. install all of the explicitly requested packages and their dependencies, then get a list of the packages that were installed, and use that list to install the complementary packages. This has been implemented by using a list of globs which should make it easier to extend in future. The previous locale package installation code assumed that the locale packages did not have any dependencies that were not already installed; now that we are installing non-locale packages this is no longer correct. In practice only the rpm backend actually made use of this assumption, so it needed to be changed to call into the existing package backend code to do the complementary package installation rather than calling rpm directly. This fixes the doc-pkgs IMAGE_FEATURES feature to work correctly, and also ensures that all dev/dbg packages get installed for dev-pkgs/dbg-pkgs respectively even if the dependency chains between those packages was not ensuring that already. The code has also been adapted to work correctly with the new SDK-from-image functionality. To that end, an SDKIMAGE_FEATURES variable has been added to allow specifying what extra image features should go into the SDK (extra, because by virtue of installing all of the packages in the image into the target part of the SDK, we already include all of IMAGE_FEATURES) with a default value of "dev-pkgs dbg-pkgs". Fixes [YOCTO #2614]. (From OE-Core rev: 72d1048a8381fa4a8c4c0d082047536727b4be47) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-07-09 13:15:08 +00:00
def fwdpkgdata(pkgn):
bitbake.conf/package: Collapse PKGDATA_DIR into a single machine specific directory Currently we have a hierarchy of pkgdata directories and the code has to put together a search path and look through each in turn until it finds the data it needs. This has lead to a number of hardcoded paths and file globing which is unpredictable and undesirable. Worse, certain tricks that should be easy like a GL specific package architecture become problematic with the curretn search paths. With the modern sstate code, we can do better and construct a single pkgdata directory for each machine in just the same way as we do for the sysroot. This is already tried and well tested. With such a single directory, all the code that iterated through multiple pkgdata directories and simply be removed and give a significant simplification of the code. Even existing build directories adapt to the change well since the package contents doesn't change, just the location they're installed to and the stamp for them. The only complication is the we need a different shlibs directory for each multilib. These are only used by package.bbclass and the simple fix is to add MLPREFIX to the shlib directory name. This means the multilib packages will repackage and the sstate checksum will change but an existing build directory will adapt to the changes safely. It is close to release however I believe the benefits this patch give us are worth consideration for inclusion and give us more options for dealing with problems like the GL one. It also sets the ground work well for shlibs improvements in 1.6. (From OE-Core rev: 1b8e4abd2d9c0901d38d89d0f944fe1ffd019379) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-09-13 12:35:31 +00:00
return os.path.join(pkgdata_dir, "runtime", pkgn)
Rework installation of dev, dbg, doc, and locale packages Use a similar mechanism that was previously used to install locales at rootfs generation time to install other "complementary" packages (e.g. *-dev packages) - i.e. install all of the explicitly requested packages and their dependencies, then get a list of the packages that were installed, and use that list to install the complementary packages. This has been implemented by using a list of globs which should make it easier to extend in future. The previous locale package installation code assumed that the locale packages did not have any dependencies that were not already installed; now that we are installing non-locale packages this is no longer correct. In practice only the rpm backend actually made use of this assumption, so it needed to be changed to call into the existing package backend code to do the complementary package installation rather than calling rpm directly. This fixes the doc-pkgs IMAGE_FEATURES feature to work correctly, and also ensures that all dev/dbg packages get installed for dev-pkgs/dbg-pkgs respectively even if the dependency chains between those packages was not ensuring that already. The code has also been adapted to work correctly with the new SDK-from-image functionality. To that end, an SDKIMAGE_FEATURES variable has been added to allow specifying what extra image features should go into the SDK (extra, because by virtue of installing all of the packages in the image into the target part of the SDK, we already include all of IMAGE_FEATURES) with a default value of "dev-pkgs dbg-pkgs". Fixes [YOCTO #2614]. (From OE-Core rev: 72d1048a8381fa4a8c4c0d082047536727b4be47) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-07-09 13:15:08 +00:00
def readpn(pkgdata_file):
pn = ""
with open(pkgdata_file, 'r') as f:
for line in f:
if line.startswith("PN:"):
pn = line.split(': ')[1].rstrip()
return pn
def readrenamed(pkgdata_file):
renamed = ""
pn = os.path.basename(pkgdata_file)
with open(pkgdata_file, 'r') as f:
for line in f:
if line.startswith("PKG_%s:" % pn):
renamed = line.split(': ')[1].rstrip()
return renamed
# Main processing loop
for g in globs:
mappedpkg = ""
# First just try substitution (i.e. packagename -> packagename-dev)
newpkg = g.replace("*", pkg)
revlink = revpkgdata(newpkg)
if os.path.exists(revlink):
mappedpkg = os.path.basename(os.readlink(revlink))
fwdfile = fwdpkgdata(mappedpkg)
if os.path.exists(fwdfile):
mappedpkg = readrenamed(fwdfile)
if not os.path.exists(fwdfile + ".packaged"):
mappedpkg = ""
Rework installation of dev, dbg, doc, and locale packages Use a similar mechanism that was previously used to install locales at rootfs generation time to install other "complementary" packages (e.g. *-dev packages) - i.e. install all of the explicitly requested packages and their dependencies, then get a list of the packages that were installed, and use that list to install the complementary packages. This has been implemented by using a list of globs which should make it easier to extend in future. The previous locale package installation code assumed that the locale packages did not have any dependencies that were not already installed; now that we are installing non-locale packages this is no longer correct. In practice only the rpm backend actually made use of this assumption, so it needed to be changed to call into the existing package backend code to do the complementary package installation rather than calling rpm directly. This fixes the doc-pkgs IMAGE_FEATURES feature to work correctly, and also ensures that all dev/dbg packages get installed for dev-pkgs/dbg-pkgs respectively even if the dependency chains between those packages was not ensuring that already. The code has also been adapted to work correctly with the new SDK-from-image functionality. To that end, an SDKIMAGE_FEATURES variable has been added to allow specifying what extra image features should go into the SDK (extra, because by virtue of installing all of the packages in the image into the target part of the SDK, we already include all of IMAGE_FEATURES) with a default value of "dev-pkgs dbg-pkgs". Fixes [YOCTO #2614]. (From OE-Core rev: 72d1048a8381fa4a8c4c0d082047536727b4be47) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-07-09 13:15:08 +00:00
else:
revlink = revpkgdata(pkg)
if os.path.exists(revlink):
# Check if we can map after undoing the package renaming (by resolving the symlink)
origpkg = os.path.basename(os.readlink(revlink))
newpkg = g.replace("*", origpkg)
Rework installation of dev, dbg, doc, and locale packages Use a similar mechanism that was previously used to install locales at rootfs generation time to install other "complementary" packages (e.g. *-dev packages) - i.e. install all of the explicitly requested packages and their dependencies, then get a list of the packages that were installed, and use that list to install the complementary packages. This has been implemented by using a list of globs which should make it easier to extend in future. The previous locale package installation code assumed that the locale packages did not have any dependencies that were not already installed; now that we are installing non-locale packages this is no longer correct. In practice only the rpm backend actually made use of this assumption, so it needed to be changed to call into the existing package backend code to do the complementary package installation rather than calling rpm directly. This fixes the doc-pkgs IMAGE_FEATURES feature to work correctly, and also ensures that all dev/dbg packages get installed for dev-pkgs/dbg-pkgs respectively even if the dependency chains between those packages was not ensuring that already. The code has also been adapted to work correctly with the new SDK-from-image functionality. To that end, an SDKIMAGE_FEATURES variable has been added to allow specifying what extra image features should go into the SDK (extra, because by virtue of installing all of the packages in the image into the target part of the SDK, we already include all of IMAGE_FEATURES) with a default value of "dev-pkgs dbg-pkgs". Fixes [YOCTO #2614]. (From OE-Core rev: 72d1048a8381fa4a8c4c0d082047536727b4be47) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-07-09 13:15:08 +00:00
fwdfile = fwdpkgdata(newpkg)
if os.path.exists(fwdfile):
mappedpkg = readrenamed(fwdfile)
else:
# That didn't work, so now get the PN, substitute that, then map in the other direction
pn = readpn(revlink)
newpkg = g.replace("*", pn)
fwdfile = fwdpkgdata(newpkg)
if os.path.exists(fwdfile):
mappedpkg = readrenamed(fwdfile)
if not os.path.exists(fwdfile + ".packaged"):
mappedpkg = ""
Rework installation of dev, dbg, doc, and locale packages Use a similar mechanism that was previously used to install locales at rootfs generation time to install other "complementary" packages (e.g. *-dev packages) - i.e. install all of the explicitly requested packages and their dependencies, then get a list of the packages that were installed, and use that list to install the complementary packages. This has been implemented by using a list of globs which should make it easier to extend in future. The previous locale package installation code assumed that the locale packages did not have any dependencies that were not already installed; now that we are installing non-locale packages this is no longer correct. In practice only the rpm backend actually made use of this assumption, so it needed to be changed to call into the existing package backend code to do the complementary package installation rather than calling rpm directly. This fixes the doc-pkgs IMAGE_FEATURES feature to work correctly, and also ensures that all dev/dbg packages get installed for dev-pkgs/dbg-pkgs respectively even if the dependency chains between those packages was not ensuring that already. The code has also been adapted to work correctly with the new SDK-from-image functionality. To that end, an SDKIMAGE_FEATURES variable has been added to allow specifying what extra image features should go into the SDK (extra, because by virtue of installing all of the packages in the image into the target part of the SDK, we already include all of IMAGE_FEATURES) with a default value of "dev-pkgs dbg-pkgs". Fixes [YOCTO #2614]. (From OE-Core rev: 72d1048a8381fa4a8c4c0d082047536727b4be47) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-07-09 13:15:08 +00:00
else:
# Package doesn't even exist...
if debug:
print "%s is not a valid package!" % (pkg)
break
if mappedpkg:
if debug:
print "%s (%s) -> %s" % (pkg, g, mappedpkg)
mappedpkgs.add(mappedpkg)
else:
if debug:
print "%s (%s) -> ?" % (pkg, g)
if debug:
print "------"
print("\n".join(mappedpkgs))
def read_value(args, usage, debug=False):
bitbake.conf/package: Collapse PKGDATA_DIR into a single machine specific directory Currently we have a hierarchy of pkgdata directories and the code has to put together a search path and look through each in turn until it finds the data it needs. This has lead to a number of hardcoded paths and file globing which is unpredictable and undesirable. Worse, certain tricks that should be easy like a GL specific package architecture become problematic with the curretn search paths. With the modern sstate code, we can do better and construct a single pkgdata directory for each machine in just the same way as we do for the sysroot. This is already tried and well tested. With such a single directory, all the code that iterated through multiple pkgdata directories and simply be removed and give a significant simplification of the code. Even existing build directories adapt to the change well since the package contents doesn't change, just the location they're installed to and the stamp for them. The only complication is the we need a different shlibs directory for each multilib. These are only used by package.bbclass and the simple fix is to add MLPREFIX to the shlib directory name. This means the multilib packages will repackage and the sstate checksum will change but an existing build directory will adapt to the changes safely. It is close to release however I believe the benefits this patch give us are worth consideration for inclusion and give us more options for dealing with problems like the GL one. It also sets the ground work well for shlibs improvements in 1.6. (From OE-Core rev: 1b8e4abd2d9c0901d38d89d0f944fe1ffd019379) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-09-13 12:35:31 +00:00
if len(args) < 3:
usage()
sys.exit(1)
pkgdata_dir = args[0]
bitbake.conf/package: Collapse PKGDATA_DIR into a single machine specific directory Currently we have a hierarchy of pkgdata directories and the code has to put together a search path and look through each in turn until it finds the data it needs. This has lead to a number of hardcoded paths and file globing which is unpredictable and undesirable. Worse, certain tricks that should be easy like a GL specific package architecture become problematic with the curretn search paths. With the modern sstate code, we can do better and construct a single pkgdata directory for each machine in just the same way as we do for the sysroot. This is already tried and well tested. With such a single directory, all the code that iterated through multiple pkgdata directories and simply be removed and give a significant simplification of the code. Even existing build directories adapt to the change well since the package contents doesn't change, just the location they're installed to and the stamp for them. The only complication is the we need a different shlibs directory for each multilib. These are only used by package.bbclass and the simple fix is to add MLPREFIX to the shlib directory name. This means the multilib packages will repackage and the sstate checksum will change but an existing build directory will adapt to the changes safely. It is close to release however I believe the benefits this patch give us are worth consideration for inclusion and give us more options for dealing with problems like the GL one. It also sets the ground work well for shlibs improvements in 1.6. (From OE-Core rev: 1b8e4abd2d9c0901d38d89d0f944fe1ffd019379) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-09-13 12:35:31 +00:00
var = args[1]
packages = args[2].split()
if not os.path.exists(pkgdata_dir):
print('ERROR: Unable to find pkgdata directory %s' % pkgdata_dir)
sys.exit(1)
def readvar(pkgdata_file, var):
val = ""
with open(pkgdata_file, 'r') as f:
for line in f:
if line.startswith(var + ":"):
val = line.split(': ')[1].rstrip()
return val
if debug:
bitbake.conf/package: Collapse PKGDATA_DIR into a single machine specific directory Currently we have a hierarchy of pkgdata directories and the code has to put together a search path and look through each in turn until it finds the data it needs. This has lead to a number of hardcoded paths and file globing which is unpredictable and undesirable. Worse, certain tricks that should be easy like a GL specific package architecture become problematic with the curretn search paths. With the modern sstate code, we can do better and construct a single pkgdata directory for each machine in just the same way as we do for the sysroot. This is already tried and well tested. With such a single directory, all the code that iterated through multiple pkgdata directories and simply be removed and give a significant simplification of the code. Even existing build directories adapt to the change well since the package contents doesn't change, just the location they're installed to and the stamp for them. The only complication is the we need a different shlibs directory for each multilib. These are only used by package.bbclass and the simple fix is to add MLPREFIX to the shlib directory name. This means the multilib packages will repackage and the sstate checksum will change but an existing build directory will adapt to the changes safely. It is close to release however I believe the benefits this patch give us are worth consideration for inclusion and give us more options for dealing with problems like the GL one. It also sets the ground work well for shlibs improvements in 1.6. (From OE-Core rev: 1b8e4abd2d9c0901d38d89d0f944fe1ffd019379) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-09-13 12:35:31 +00:00
print "read-value('%s', '%s' '%s'" % (pkgdata_dir, var, packages)
for package in packages:
pkg_split = package.split('_')
pkg_name = pkg_split[0]
if debug:
print "package: '%s'" % pkg_name
bitbake.conf/package: Collapse PKGDATA_DIR into a single machine specific directory Currently we have a hierarchy of pkgdata directories and the code has to put together a search path and look through each in turn until it finds the data it needs. This has lead to a number of hardcoded paths and file globing which is unpredictable and undesirable. Worse, certain tricks that should be easy like a GL specific package architecture become problematic with the curretn search paths. With the modern sstate code, we can do better and construct a single pkgdata directory for each machine in just the same way as we do for the sysroot. This is already tried and well tested. With such a single directory, all the code that iterated through multiple pkgdata directories and simply be removed and give a significant simplification of the code. Even existing build directories adapt to the change well since the package contents doesn't change, just the location they're installed to and the stamp for them. The only complication is the we need a different shlibs directory for each multilib. These are only used by package.bbclass and the simple fix is to add MLPREFIX to the shlib directory name. This means the multilib packages will repackage and the sstate checksum will change but an existing build directory will adapt to the changes safely. It is close to release however I believe the benefits this patch give us are worth consideration for inclusion and give us more options for dealing with problems like the GL one. It also sets the ground work well for shlibs improvements in 1.6. (From OE-Core rev: 1b8e4abd2d9c0901d38d89d0f944fe1ffd019379) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-09-13 12:35:31 +00:00
revlink = os.path.join(pkgdata_dir, "runtime-reverse", pkg_name)
if debug:
print(revlink)
if os.path.exists(revlink):
mappedpkg = os.path.basename(os.readlink(revlink))
qvar = var
if qvar == "PKGSIZE":
# append packagename
qvar = "%s_%s" % (var, mappedpkg)
print(readvar(revlink, qvar))
Rework installation of dev, dbg, doc, and locale packages Use a similar mechanism that was previously used to install locales at rootfs generation time to install other "complementary" packages (e.g. *-dev packages) - i.e. install all of the explicitly requested packages and their dependencies, then get a list of the packages that were installed, and use that list to install the complementary packages. This has been implemented by using a list of globs which should make it easier to extend in future. The previous locale package installation code assumed that the locale packages did not have any dependencies that were not already installed; now that we are installing non-locale packages this is no longer correct. In practice only the rpm backend actually made use of this assumption, so it needed to be changed to call into the existing package backend code to do the complementary package installation rather than calling rpm directly. This fixes the doc-pkgs IMAGE_FEATURES feature to work correctly, and also ensures that all dev/dbg packages get installed for dev-pkgs/dbg-pkgs respectively even if the dependency chains between those packages was not ensuring that already. The code has also been adapted to work correctly with the new SDK-from-image functionality. To that end, an SDKIMAGE_FEATURES variable has been added to allow specifying what extra image features should go into the SDK (extra, because by virtue of installing all of the packages in the image into the target part of the SDK, we already include all of IMAGE_FEATURES) with a default value of "dev-pkgs dbg-pkgs". Fixes [YOCTO #2614]. (From OE-Core rev: 72d1048a8381fa4a8c4c0d082047536727b4be47) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-07-09 13:15:08 +00:00
def lookup_pkg(args, usage, debug=False):
if len(args) < 2:
usage()
sys.exit(1)
pkgdata_dir = args[0]
pkgs = args[1].split()
if not os.path.exists(pkgdata_dir):
print('ERROR: Unable to find pkgdata directory %s' % pkgdata_dir)
sys.exit(1)
mappings = defaultdict(list)
for pkg in pkgs:
pkgfile = os.path.join(pkgdata_dir, 'runtime', pkg)
if os.path.exists(pkgfile):
with open(pkgfile, 'r') as f:
for line in f:
fields = line.rstrip().split(': ')
if fields[0] == 'PKG_%s' % pkg:
mappings[pkg].append(fields[1])
break
if len(mappings) < len(pkgs):
missing = list(set(pkgs) - set(mappings.keys()))
sys.stderr.write("ERROR: the following packages could not be found: %s\n" % ', '.join(missing))
sys.exit(1)
items = []
for pkg in pkgs:
items.extend(mappings.get(pkg, []))
print '\n'.join(items)
def lookup_recipe(args, usage, debug=False):
if len(args) < 2:
usage()
sys.exit(1)
pkgdata_dir = args[0]
pkgs = args[1].split()
if not os.path.exists(pkgdata_dir):
print('ERROR: Unable to find pkgdata directory %s' % pkgdata_dir)
sys.exit(1)
mappings = defaultdict(list)
for pkg in pkgs:
pkgfile = os.path.join(pkgdata_dir, 'runtime-reverse', pkg)
if os.path.exists(pkgfile):
with open(pkgfile, 'r') as f:
for line in f:
fields = line.rstrip().split(': ')
if fields[0] == 'PN':
mappings[pkg].append(fields[1])
break
if len(mappings) < len(pkgs):
missing = list(set(pkgs) - set(mappings.keys()))
sys.stderr.write("ERROR: the following packages could not be found: %s\n" % ', '.join(missing))
sys.exit(1)
items = []
for pkg in pkgs:
items.extend(mappings.get(pkg, []))
print '\n'.join(items)
def find_path(args, usage, debug=False):
if len(args) < 2:
usage()
sys.exit(1)
pkgdata_dir = args[0]
targetpath = args[1]
if not os.path.exists(pkgdata_dir):
print('ERROR: Unable to find pkgdata directory %s' % pkgdata_dir)
sys.exit(1)
import json
import fnmatch
for root, dirs, files in os.walk(os.path.join(pkgdata_dir, 'runtime')):
for fn in files:
with open(os.path.join(root,fn)) as f:
for line in f:
if line.startswith('FILES_INFO:'):
val = line.split(':', 1)[1].strip()
dictval = json.loads(val)
for fullpth in dictval.keys():
if fnmatch.fnmatchcase(fullpth, targetpath):
print("%s: %s" % (fn, fullpth))
break
def main():
parser = optparse.OptionParser(
usage = '''%prog [options] <command> <arguments>
Available commands:
glob <pkgdatadir> <pkglistfile> "<globs>"
expand one or more glob expressions over the packages listed in
pkglistfile (one package per line)
lookup-pkg <pkgdatadir> "<recipe-pkgs>"
look up the specified recipe-space package name(s) to see what the
final runtime package name is (e.g. eglibc becomes libc6)
lookup-recipe <pkgdatadir> "<pkgs>"
look up the specified package(s) to see which recipe they were
produced by
find-path <pkgdatadir> <path>
find the package providing the specified path (wildcards * ? allowed)
read-value <pkgdatadir> <value-name> "<pkgs>"
read the named value from the pkgdata files for the specified
packages''')
parser.add_option("-d", "--debug",
help = "Report all SRCREV values, not just ones where AUTOREV has been used",
action="store_true", dest="debug", default=False)
options, args = parser.parse_args(sys.argv)
args = args[1:]
if len(args) < 1:
parser.print_help()
sys.exit(1)
if args[0] == "glob":
glob(args[1:], parser.print_help, options.debug)
elif args[0] == "lookup-pkg":
lookup_pkg(args[1:], parser.print_help, options.debug)
elif args[0] == "lookup-recipe":
lookup_recipe(args[1:], parser.print_help, options.debug)
elif args[0] == "find-path":
find_path(args[1:], parser.print_help, options.debug)
elif args[0] == "read-value":
read_value(args[1:], parser.print_help, options.debug)
Rework installation of dev, dbg, doc, and locale packages Use a similar mechanism that was previously used to install locales at rootfs generation time to install other "complementary" packages (e.g. *-dev packages) - i.e. install all of the explicitly requested packages and their dependencies, then get a list of the packages that were installed, and use that list to install the complementary packages. This has been implemented by using a list of globs which should make it easier to extend in future. The previous locale package installation code assumed that the locale packages did not have any dependencies that were not already installed; now that we are installing non-locale packages this is no longer correct. In practice only the rpm backend actually made use of this assumption, so it needed to be changed to call into the existing package backend code to do the complementary package installation rather than calling rpm directly. This fixes the doc-pkgs IMAGE_FEATURES feature to work correctly, and also ensures that all dev/dbg packages get installed for dev-pkgs/dbg-pkgs respectively even if the dependency chains between those packages was not ensuring that already. The code has also been adapted to work correctly with the new SDK-from-image functionality. To that end, an SDKIMAGE_FEATURES variable has been added to allow specifying what extra image features should go into the SDK (extra, because by virtue of installing all of the packages in the image into the target part of the SDK, we already include all of IMAGE_FEATURES) with a default value of "dev-pkgs dbg-pkgs". Fixes [YOCTO #2614]. (From OE-Core rev: 72d1048a8381fa4a8c4c0d082047536727b4be47) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-07-09 13:15:08 +00:00
else:
parser.print_help()
sys.exit(1)
if __name__ == "__main__":
main()