Commit Graph

41056 Commits

Author SHA1 Message Date
brian avery 64b8e17892 toaster-manual: Switch to pip3 dependency installer
Our switch to python3 requires us to dump the virtualenv approach
as long as we need python to resolve to python2 in the same shell
that python3 resolves to python3. To enable users to install
Toaster's python package dependencies locally, we are changing to
using pip3 and these doc changes explain how to do that.

(From yocto-docs rev: 06b8820bcb90d1855ccb2712b616bb89847a148f)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 15:04:49 +01:00
Ed Bartosh f6378c10ef toaster-manual: add instructions to create user
Creating separate toaster account would make production setup
more isolated and maintainable.

(From yocto-docs rev: 33c638d3e9a68d06d929276c6064d567ed7fe696)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 15:04:49 +01:00
Scott Rifenbark 444a80a355 ref-manual: Added a new section on setscene task variants
Fixes [YOCTO #9251]

I added a new section on the task variants for setscene, which uses
the shared state cache.  The section describes the BitBake processing
that helps the build system to not have to build everything from scratch.

(From yocto-docs rev: 6aa0e6b837de14dd7d6c5ef34328bbf268dd6d71)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 15:04:49 +01:00
Scott Rifenbark 4fb1961acf ref-manual: Fixed typo in the "Shared State" section.
Fixes [YOCTO #9823]

The do_deploy[sstate-inputdirs] string was wrongly
do_deploy[sstate-inputsdirs].

(From yocto-docs rev: 49e7cb239f4f4f70dc5738686f1bca135772bcbe)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 15:04:49 +01:00
Scott Rifenbark b67f865e67 ref-manual: Review edits to the PR variable in glossary.
Fixes [YOCTO #9843]

Some minor rewordings and removal of a stray comma.

(From yocto-docs rev: 0b0c8b0b1c4edb86eb2b20c73ffb83d640f6b916)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 15:04:49 +01:00
Scott Rifenbark e5fabb90dc ref-manual: Updated the RDEPENDS variable description in the glossary
Fixes [YOCTO #9380]

Updated the shlibdeps description for this variable to try and
satisfy automatically added version restrictions.

(From yocto-docs rev: d94de0b24591206620c83a9022a9d813141a7637)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 15:04:49 +01:00
Scott Rifenbark b66e84e19d ref-manual: Updated the PR variable description.
Fixes [YOCTO #9843]

The variable description was very brief.  These changes added some
substance to the description and how the OpenEmbedded build system
uses the variable.

(From yocto-docs rev: cc9117d4e16c56dd54f81804fda94561e61d91a7)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 15:04:49 +01:00
Scott Rifenbark 33e1c39361 dev-manual: Review edits to the package installation section
Fixes [YOCTO #9672]

A couple typos here needed fixed.  Also, a missing statement in
the JSON example.

(From yocto-docs rev: 8756cb363a70a8b77077a666c62c0f678ffb3d3e)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 15:04:48 +01:00
Scott Rifenbark 77031ba572 ref-manual: Updated the flag descriptions for shared state details
Fixes [YOCTO 9823]

I added more details to the explanations of how shared state is
implemented.  Included a bulleted list of the various statements
of code to help explain flags and settings.

(From yocto-docs rev: 518352f88c8dda16f2915a7bb9901ffd7686d739)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 15:04:48 +01:00
Scott Rifenbark 57acc32724 dev-manual: Edits to the package feed creation section.
Updated the introduction of the trio of variables used for package
feed naming in the "Build Considerations" section.

Fixes [YOCTO #1882]

(From yocto-docs rev: 4c7efd08f9f543dd3c4f2e6d68c72d16fd7b0b8c)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 15:04:48 +01:00
Scott Rifenbark fc518b5a48 ref-manual: Updated the DISTRO_FEATURES description of Bluez5
Edits to explain that by default, DISTRO_FEATURES backfills
bluetooth support with Bluez5.  If the user wants to use the
Bluez4 feature, they need to backfill consider Bluez5.

(From yocto-docs rev: 4e8b34c8c5841cdc1cae564decae4689ae4f2b11)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 15:04:48 +01:00
Paul Eggleton 45baa90c8c bitbake: fetch2: implement progress support
Implement progress reporting support specifically for the fetchers. For
fetch tasks we don't necessarily know which fetcher will be used (we
might initially be fetching a git:// URI, but if we instead download a
mirror tarball we may fetch that over http using wget). These programs
also have different abilities as far as reporting progress goes (e.g.
wget gives us percentage complete and rate, git gives this some of the
time depending on what stage it's at). Additionally we filter out the
progress output before it makes it to the logs, in order to prevent the
logs filling up with junk.

At the moment this is only implemented for the wget and git fetchers
since they are the most commonly used (and svn doesn't seem to support
any kind of progress output, at least not without doing a relatively
expensive remote file listing first).

Line changes such as the ones you get in git's output as it progresses
don't make it to the log files, you only get the final state of the line
so the logs aren't filled with progress information that's useless after
the fact.

Part of the implementation for [YOCTO #5383].

(Bitbake rev: 4027649f422ee64b1c4e1ad8d48ac295050afbff)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 15:04:37 +01:00
Elliot Smith 843f7ae410 toaster.bbclass: only scan files-in-image.txt if it exists
We can reach the method in toaster.bbclass which tries to read from
the files-in-image.txt file via a build which doesn't create that
file (e.g. "bitbake core-image-minimal -c rootfs"). This causes
the build to fail with an exception.

Check that this file exists before trying to read from it.

[YOCTO #9784]

(From OE-Core rev: 8b369cdd73ab17cdf834a591b97b25840caeb740)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 09:03:32 +01:00
Elliot Smith ac339ece20 toaster.bbclass: remove directory scan logic for detecting artifacts
toaster.bbclass does a scan of the image deploy and SDK directories
when a build finishes. However, this brings no benefit and could be
better managed and made easier to modify if moved to toasterui and
carried out when the BuildCompleted event occurs.

Remove the image scan code from toaster.bbclass, prior to moving it
to toasterui and buildinfohelper.

Also remove the license manifest update code, as this can also be
done from toasterui.

The postfuncs for do_populate_sdk are retained, but no longer
do the directory scan for SDK artifacts. Instead, they fire
an event with the value of the TOOLCHAIN_OUTPUTNAME variable,
as this is only accessible at the point when the do_populate_sdk
and do_populate_sdk_ext tasks are run. The value of this can then
be used by buildinfohelper to find the SDK artifacts produced by a
target.

[YOCTO #9002]

(From OE-Core rev: 67ebb5406c0fcdd1b28bf446249aa6fe34a741a8)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 09:03:32 +01:00
Elliot Smith 8d08a73225 bitbake: toaster-tests: package count/size shouldn't show for non-image builds
If a build doesn't produce any image files, the package count
and size shouldn't be shown.

Also add some metadata to build dashboard elements so it is clear
what they're for, and so they can be queried by the tests.

(Bitbake rev: 3d5090af4475b1d0bb56911a8e30abf9097c1b3c)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:52 +01:00
Elliot Smith 8ccf8392aa bitbake: toaster-tests: new custom image button shouldn't show for non-image builds
[YOCTO #9514]

(Bitbake rev: 9bc014d89434400c1493fa9f07ce3a51d37dab51)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:52 +01:00
Elliot Smith db5426b079 bitbake: toaster-tests: add tests for build artifact display on build dashboard
Add tests for display of image, kernel and SDK artifacts on the
build dashboard, checking that the "Images" option in the left-hand
menu and the "Build artifacts" section display correctly for
different types of build.

Also add metadata to elements on the build dashboard so it's clearer
what they represent, and to assist in finding them in the tests.

Add a method to the test helper to make it more convenient to check
whether a single element matching a selector exists.

[YOCTO #8556]
[YOCTO #8563]
[YOCTO #9500]

(Bitbake rev: 644a888ce5a2141f2e6e1c22430e196b65cb1313)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:52 +01:00
Elliot Smith 9475a684c4 bitbake: toaster: the customise image button shouldn't rely on targets
The build dashboard customise image button (for creating a new
custom image based on an image recipe used by a build) shouldn't
rely on targets: whether a new custom image can be created or not
depends on whether any of the recipes used by the build are image
recipes.

Modify the method used to determine whether a build has customisable
images to look at the image recipes used during the build, rather
than whether the targets run by the build refer to image recipes.

(Bitbake rev: 6648876c91134bda8498b4f8d7ace9147ec0d985)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:52 +01:00
Elliot Smith 2db40e3771 bitbake: toaster: add package manifest path to Target objects
Store the path to the *.rootfs.manifest file for targets which
generate images.

A link to the package manifest is displayed in the build dashboard
for targets which produce image files.

Like the license manifest path, if a target would have produced
the package manifest (but didn't, because it already existed), that
path is copied from the target which did produce the package
manifest.

(Bitbake rev: 79b8e349a0da2ea6b97ad82daa5837e6dfffe0af)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:52 +01:00
Elliot Smith 1027e0e313 bitbake: toaster: better display of targets which produced no images
SDK targets (populate_sdk) produce SDK artifacts but no image files.
Currently, these targets appear under the "Images" heading in the
build dashboard, even though they aren't strictly image targets.

Change the heading to "Build artifacts". Also remove the section
which states that a build produced no image files: this is not
correct for populate_sdk targets (those targets don't produce
image files under any circumstances); and other changes mean
that all targets which do produce images will now show those
files.

The check for whether to display the "Build artifacts" section also
needs to change, as we show targets here which didn't produce any
images but did produce SDK artifacts.

[YOCTO #8556]

(Bitbake rev: b4dce68045c4615e7a6a474e952f670721a3b54e)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:52 +01:00
Elliot Smith adbf206526 bitbake: toaster: show manifests in their own section of the build dashboard
In the build dashboard, add a section for manifests to each target,
which will contain the license and package manifests for a target.

As we don't record the package manifest (yet), just move the
license manifest link to that section for now.

(Bitbake rev: 58cb685b567a081834a605530957f165d35fcf5e)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:52 +01:00
Elliot Smith 07a58a8944 bitbake: toaster: only show "New custom image" button for builds with image targets
Add a has_image_targets() method to Build, and use that to hide
the "New custom image" button on the build dashboard if a build
has no targets which build images.

[YOCTO #9514]

(Bitbake rev: 3c4b053e44ea512ef2ced67289a7b0161db6ce9b)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:52 +01:00
Elliot Smith 587275eefd bitbake: toaster: use has_images() methods to display images correctly
In the build dashboard, we had issues with showing images correctly,
as we were using the is_image property of targets to determine
whether a target would have image files. This property can
be set to True if a target refers to an image recipe
(e.g. "core-image-minimal"), even if the task used in the build
didn't produce any image files.

By adding has_images() methods to the Target and Build objects,
which count associated Target_Image_File objects,
we can correctly determine whether a target has image files
associated with it, and if any of the targets for a build has
image files. This means that we can screen out the left-hand
"Images" menu options for builds which contained image-related
targets (e.g. "core-image-minimal") but which didn't produce
any images (e.g. "rootfs" task).

[YOCTO #9500]
[YOCTO #9784]

(Bitbake rev: f6bba0ff254d5ed3163151d4b938f3a43c9acb0a)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00
Elliot Smith 150e5588a0 bitbake: buildinfohelper: only record image files for tasks which make images
If a target is built which is classified as an "image" target
(e.g. "core-image-minimal"), Toaster reads the list of files in
the image (from the files-in-image.txt file).

However, Toaster continues to do this for builds which don't
produce images, if the recipe providing the target is an
image recipe. This can result in a list of files in the image
being attached to a target which didn't produce an image (e.g.
rootfs).

When associating files with an image, ensure that only targets
with a task which produces an image have "files in the image"
associated with them.

[YOCTO #9784]

(Bitbake rev: 44375d0c2a88e0070b8067c9285b89c54eaf3152)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00
Elliot Smith 00c2c0be5e bitbake: toaster: improve scan for SDK artifacts
SDK artifacts were previously picked up by toaster.bbclass and
notified to buildinfohelper (via toasterui). The artifacts
were then added to the Build object, so that it wasn't clear
which artifact went with which target; we were also unable
to attach SDK artifacts to a Build if they had already been
attached to a previous build.

Now, toaster.bbclass just notifies the TOOLCHAIN_OUTPUTNAME when
a populate_sdk* target completes. The scan is moved to buildinfohelper,
where we search the SDK deploy directory for files matching
TOOLCHAIN_OUTPUTNAME and attach them to targets (not builds).

If an SDK file is not produced by a target, we now look for a
similar, previously-run target which did produce artifacts.
If there is one, we clone the SDK artifacts from that target
onto the current one.

This all means that we can show SDK artifacts by target, and should
always get artifacts associated with a target, regardless of whether
it really build them.

This requires an additional model, TargetSDKFile, which tracks
the size and path of SDK artifact files with respect to Target
objects.

[YOCTO #8556]

(Bitbake rev: 5e650c611605507e1e0d1588cd5eb6535c2d34fc)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00
Elliot Smith f39ae146ea bitbake: buildinfohelper: fix retrieval of targets
When buildinfohelper records the targets for a build, it looks
up any existing targets for a build and creates them if they
are not present. This is because in the case of Toaster-triggered
builds, the Target objects have already been created (inside
triggerBuild()) and don't need to be recreated; but in the case
of cli builds, the Target objects have to be created by
buildinfohelper.

The issue is that the code for retrieving an existing target for
a build only looks for Targets with a matching target and build,
e.g. Targets for build X with target "core-image-minimal". But it
is perfectly legitimate to call bitbake with a command like
"bitbake core-image-minimal:do_populate_sdk
core-image-minimal:do_populate_sdk_ext". In such a case, the
code which looks for matching targets finds two objects, as it
doesn't filter by task.

Add the task into the filter for the Target so that only one
Target object is be returned. Note that a command
line like "bitbake recipe:task recipe:task" will still cause an
error as bitbake doesn't de-duplicate the command line arguments
and will run the recipe:task combination twice.

(Bitbake rev: 1c0a689fdaae6469d4afb98583161073d32ea50b)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00
Elliot Smith 4125da7763 bitbake: toaster: attach kernel artifacts to targets
The bzImage and modules files were previously attached to a build,
rather than to the target which produced them. This meant it was
not possible to determine which kernel artifact produced by a
build came from which target; which in turn made it difficult to
associate existing kernel artifact with targets when those
targets didn't produce artifacts (e.g. if the same machine + target
combination was built again and didn't produce a bzImage or modules
file because those files already existed).

By associating kernel artifacts with the target (via a new
TargetArtifactFile model), we make it possible to find all
the artifacts for a given machine + target combination. Then, in
cases where a build is completed but its targets don't produce
any artifacts, we can find a previous Target object with the same
machine + target and copy its artifacts to the targets for a
just-completed build.

Note that this doesn't cover SDK artifacts yet, which are still
retrieved in toaster.bbclass and show up as "Other artifacts",
lumped together for the whole build rather than by target.

[YOCTO #8556]

(Bitbake rev: 9b151416e428c2565a27d89116439f9a8d578e3d)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00
Elliot Smith e9808576da bitbake: toaster: improve image file suffix retrieval
Refactor retrieval of suffix from image file path, making it a
a method on Target_Image_File. This makes it easier to use this
in the build dashboard for individual images, plus reduces the
complexity of the code required to get all of the image file
suffixes for a build.

(Bitbake rev: 9c38de3dec74c122c2060cad37331bdafc6858ec)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00
Elliot Smith 5dfa120a7c bitbake: toaster: do image and artifact scan on BuildCompleted
Move the image and artifact scan code from toaster.bbclass and
consolidate its logic with the existing logic in buildinfohelper.

Remove handler setup for events which used to be fired from
toaster.bbclass but which are now handled directly by buildinfohelper.

[YOCTO #8556]

(Bitbake rev: f0085cd554604cfff4a3f40a34825fbb6878004f)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00
Elliot Smith b0585e6b0c bitbake: toaster: display Target targets in build dashboard
The build dashboard was showing the targets for the build in the page
heading and title as "Target object".

Add a filter which extracts the "target" from each Target object
as a string so that the heading and title display correctly.

Also sort the image file suffixes alphabetically.

[YOCTO #8556]

(Bitbake rev: 33a24992139e7dc05d14bbe1da60299b732945aa)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00
Ed Bartosh 844e1f9e1d bitbake: runqueue: improve exception logging
Runqueue errors direct the user to view the "failure below",
but no additional error message is available.

Log the stacktrace so that the user can see what went wrong.

Also fix a typo in the log message.

(Bitbake rev: e191f401e372ee181bc02250232ad9cb9a0e9477)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00
Robert Yang 7a220aeaae bitbake: lib/bb/utils.py: return sorted OrderedDict in explode_dep_versions2
The OrderedDict's item is sorted by insertion order, there might be a
problem when build the same recipe again, for example:
- First build of acl:
  Depends: libattr1 (>= 2.4.47), libc6 (>= 2.24)
- Second build of acl:
  Depends: libc6 (>= 2.24), libattr1 (>= 2.4.47)

They are exactly the same depends, but tools like "diff" doesn't think
so. Return sorted OrderedDict will fix the problem.

(Bitbake rev: a392f19f16ef8202ce3c12afbeb186a02438da17)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:51 +01:00
Paul Eggleton bc3a206de3 bitbake: knotty: avoid errors when fetching outside of a task
In a few places we use the fetcher code to fetch files outside of a
task, for example uninative in OE. In that case the pid of the event is
0 and that was causing an error in BBUIHelper.eventHandler(). Check the
pid and do nothing if it's 0.

(Bitbake rev: 59cb919e5cd5c653fb4d69b2d6a4320648443e10)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:50 +01:00
Ross Burton 2afcbfef2c bitbake: build: don't use $B as the default cwd for functions
When bitbake executes a shell or Python function it can cd/chdir() into a
directory before executing the task. If no directory is specified then the
default of $B is used.  However $B is an OpenEmbedded variable and BitBake
shouldn't be aware of it.

To solve this change the semantics slightly so that if no directory is
specified, the current working directory isn't changed.  There's also a sanity
check that emits a warning if a Python task does os.chdir() without restoring
the old path, and the previous working directory is restored.

This does change semantics: whereas before a function in OE would have $B as the
working directory unless specified, now the working directory is the top of the
build tree.  Any breakage this causes can be solved by either adding
do_some_task[dirs] = "${B}" or by using absolute paths in the task.

[ YOCTO #4634 ]

(Bitbake rev: 67a7b8b021badc17d8fdf447c250e79d291e75f7)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:56:50 +01:00
Ross Burton faa726824d bitbake: lib/bb/build.py: decode the command as UTF-8
The messaging FIFO is UTF-8, so decode the command as UTF-8 as well as the value
as otherwise "bberror" != b("bberror") and none of the messages from shell
functions are ever displayed.

Also add an else to the command parser so unhandled commands are noticed.

[ YOCTO #9947 ]

(Bitbake rev: 42d727743fa599e0a3c5ad2c29a1e6ede1a918bb)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:47:06 +01:00
Ross Burton b7a741cbca bitbake: lib/bb/utils: fix set_process_name
With Python 3 create_string_buffer needs a bytes() not a str() but as we were
catching all exceptions nobody noticed.

[ YOCTO #9910 ]

(Bitbake rev: 6576a9a95486c28a01d4211b4a33cc3e2c55a7cc)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 08:47:06 +01:00
George McCollister 627d01997f useradd-staticids: use map() instead of imap()
In Python3 the itertools module's imap function has been migrated to the
globalname space as map(). Calling itertools.imap() will fail because it
no longer exists.

(From OE-Core rev: da7a2c7b00b40a8759dbe9f4ab6df3e337e3d6b6)

Signed-off-by: George McCollister <george.mccollister@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-12 23:12:00 +01:00
Jussi Kukkonen fd08714c26 python3: Add compression to python3-shell dependencies
python3-shell needs python3-compression for tarfile.

(From OE-Core rev: fe5979534bd4fc1f3e5401c9a86e4aff571aec24)

Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-12 23:10:16 +01:00
Jussi Kukkonen 4473e64ba7 python3-native: Add "io" to rprovides
(From OE-Core rev: 824fa3f9a5e10348b18cf00e6f562f5ec781ac26)

Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-12 23:10:16 +01:00
Jussi Kukkonen 5e747eb3c8 python3: Add dependencies for python3-misc
http.server requires email.parser. argparse requires
codecs and textutils.

(From OE-Core rev: 64c307c8b1af32e1219e7c9ad3f634869e0fd33f)

Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-12 23:10:16 +01:00
Christopher Larson 114206fb4c buildstats-summary: round the floating point percentage
This was rounded in python 2, but python 3 changed the default behavior of /.
We could switch to the same behavior as previous by switching to // rather
than /, but there's value in keeping at least one decimal point, to avoid the
misleading case where it says 0% but the reuse is non-zero.

(From OE-Core rev: 35d36a4d097ce8a0fd0be2f795e3d5052d4f753c)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-12 23:10:16 +01:00
Paul Eggleton 029e3ebcb2 lib/oe/patch: handle encoding differences in patch files
With Python 3, the encoding of a file is significant; several recipes in
OE-Core have patches which are not fully utf-8 decodable e.g. man,
lrzsz, and gstreamer1.0-libav, leading to errors when using devtool's
modify, upgrade or extract subcommands on these recipes. To work around
this, try reading the patch file as utf-8 first and if that fails try
latin-1 before giving up.

(From OE-Core rev: 7f4d7a6f51569954e204f110827a8ce256bcdc68)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-12 23:10:16 +01:00
Paul Eggleton f465039737 recipetool: create: support specifying a file as the local source
It is currently possible to specify a file (e.g. a tarball) on the local
disk as the source, but you have to know to put file:// in front of it.
There's really no need to force users to jump through that hoop if they
really want to do this so check if the specified source is a file and
prefix it with file:// if that's the case.

Also ensure the same works for "devtool add" at the same time.

(From OE-Core rev: 71350003790c38e84b0e525a71a2fe5d24e3d083)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-12 23:10:15 +01:00
Paul Eggleton 54486a1ac3 recipetool: create: fix handling of github URLs
For a while now, Github hasn't been advertising a specific repository
URL since cloning the web URL with git works. Armed with this knowledge
and fully expecting people to just paste the github URL, we need to
handle this situation specially. If it looks like a github URL to the
root of a repository then treat it as a git repository instead of a
normal https URL to be fetched by the wget fetcher.

(From OE-Core rev: 7998dc3597657229507e5c140fceef1e485ac402)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-12 23:10:15 +01:00
Maxin B. John 2c9291dfb9 graph-tool: convert to python3
move graph-tool to python3

(From OE-Core rev: 0d0864ae0ff9e53623ad1c7146b071f2a046f21f)

Signed-off-by: Maxin B. John <maxin.john@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-12 23:10:15 +01:00
Paul Eggleton baed9bd848 scripts/contrib/devtool-stress: exclude more recipes by default
These recipes can't be used with devtool because they can't be unpacked
in the normal way.

(From OE-Core rev: b2cf098969b8b800a78d650cf60c0b5ad31c85b5)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-12 23:10:15 +01:00
Paul Eggleton 89f8348dc5 scripts/contrib/devtool-stress: skip incompatible recipes
If devtool returns exit code 4 then record the recipes as "skipped"
rather than "failed" - these are recipes we know cannot work (usually
because they don't provide any source).

(From OE-Core rev: 8fc109f1cb6eb437c12d2d11a6937de6f035e296)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-12 23:10:15 +01:00
Paul Eggleton 5c91537ab2 devtool: return specific exit code for incompatible recipes
Certain recipes cannot be used with devtool extract / modify / upgrade -
usually because they don't provide any source. Return a specific exit
code (4) so that scripts such as scripts/contrib/devtool-stress.py know
the difference between this and a genuine failure.

(From OE-Core rev: ffd295fed4ab81fc0bd00bb145ef4d72c49584bf)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-12 23:10:15 +01:00
Paul Eggleton d56caf3408 devtool: reset: allow reset to work if the recipe file has been deleted
We were attempting to open the recipe file unconditionally here - we
need to account for the possibility that the recipe file has been
deleted or moved away by the user.

(From OE-Core rev: 47822a2aff56fd338c16b5ad756feda9f395a8a1)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-12 23:10:15 +01:00
Paul Eggleton 117c80f9dc devtool: update-recipe: fix --initial-rev option
In OE-Core revision 7baf57ad896112cf2258b3e2c2a1f8b756fb39bc I changed
the default update-recipe behaviour to only update patches for commits
that were changed; unfortunately I failed to handle the --initial-rev
option which was broken after that point. Rework how the initial
revision is passed in so that it now operates correctly.

(From OE-Core rev: b2ca2523cc9e51a4759b4420b07b0b67b3f5ac43)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-12 23:10:15 +01:00