Commit Graph

154 Commits

Author SHA1 Message Date
Paul Eggleton ac5e720575 bitbake: lib: implement basic task progress support
For long-running tasks where we have some output from the task that
gives us some idea of the progress of the task (such as a percentage
complete), provide the means to scrape the output for that progress
information and show it to the user in the default knotty terminal
output in the form of a progress bar. This is implemented using a new
TaskProgress event as well as some code we can insert to do output
scanning/filtering.

Any task can fire TaskProgress events; however, if you have a shell task
whose output you wish to scan for progress information, you just need to
set the "progress" varflag on the task. This can be set to:
 * "percent" to just look for a number followed by a % sign
 * "percent:<regex>" to specify your own regex matching a percentage
   value (must have a single group which matches the percentage number)
 * "outof:<regex>" to look for the specified regex matching x out of y
   items completed (must have two groups - first group needs to be x,
   second y).
We can potentially extend this in future but this should be a good
start.

Part of the implementation for [YOCTO #5383].

(Bitbake rev: 0d275fc5b6531957a6189069b04074065bb718a0)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:26 +01:00
Robert Yang c6d50b2729 bitbake: lib/bb/build.py: remove task flag in deltask()
Otherwise the function like d.getVarFlag(e, 'task', True) which is used by
do_listtasks will still get it, and list the deleted tasks.

(Bitbake rev: 779d73619daf59f76f5b0313e7fb5409f6e82553)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-23 14:26:33 +01:00
Richard Purdie 0f2c59367a bitbake: bitbake: Convert to python 3
Various misc changes to convert bitbake to python3 which don't warrant
separation into separate commits.

(Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:02 +01:00
Richard Purdie 654eadfa30 bitbake: bitbake: Update logger.warn() -> logger.warning()
python deprecated logger.warn() in favour of logger.warning(). This is only
used in bitbake code so we may as well just translate everything to avoid
warnings under python 3. Its safe for python 2.7.

(Bitbake rev: 676a5f592e8507e81b8f748d58acfea7572f8796)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-11 10:34:30 +01:00
Christopher Larson 2d2d312188 bitbake: bb.build: handle __builtins__ as a module
Fixes pypy support.

(Bitbake rev: e90cfc655affeec8f5519f7078dad5f99db3c461)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-06 10:18:12 +01:00
Richard Purdie d73da22b0e bitbake: build/utils: Allow python functions to execute with real exception handling
With the code as it stands today it not possible to execute a python function
and get "normal" python exception handling behaviour. If a python function
raises an exception, it forces a traceback to be printed and the exception
becomes a FuncFailed exception.

This adds in a parameter 'pythonexception' which allows standard python
exceptions to be passed unchanged with no traceback. Ultimately we may want
to change to this convention in various places but at least it means we can
start to add sane functions now.

(Bitbake rev: 85cf22fd0ed26bb7dc7738ef2a10441891f38ae2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-31 09:13:04 +01:00
Richard Purdie 400f53003e bitbake: build: Improve python execution tracebacks
If something fails in a exec_func_python() the current stack trace shows
incorrect filenames and linenumbers. For example:

The stack trace of python calls that resulted in this exception/failure was:
File: '/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb', lineno: 200, function: <module>
     0196:                chksum = bb.utils.sha256_file(fn)
     0197:                f.write('%s\t%s\n' % (chksum, os.path.relpath(fn, baseoutpath)))
     0198:
     0199:
 *** 0200:copy_buildsystem(d)
     0201:
File: '/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb', lineno: 9, function: copy_buildsystem
     0005:IMAGE_FEATURES += "splash package-management x11-base x11-sato ssh-server-dropbear hwcodecs"
     0006:
     0007:LICENSE = "MIT"
     0008:
 *** 0009:inherit core-image
     0010:
     0011:IMAGE_INSTALL += "packagegroup-core-x11-sato-games"
File: '/usr/lib/python2.7/subprocess.py', lineno: 535, function: check_call
     0531:    The arguments are the same as for the Popen constructor.  Example:
     0532:
     0533:    check_call(["ls", "-l"])
     0534:    """
 *** 0535:    retcode = call(*popenargs, **kwargs)
     0536:    if retcode:
     0537:        cmd = kwargs.get("args")
     0538:        if cmd is None:
     0539:            cmd = popenargs[0]

The problem is the use of "FILE" to obtain the current filename. Instead,
we therefore inject the function being executed into the methodpool which
allows us to correct its linenumber and filename information. We can then
clearly mark the initial piece as autogenerated and the rest of the linenumber
and filename information should be correct. Afterwards the trace starts:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:copy_buildsystem(d)
     0003:
File: '/media/build1/poky/meta/classes/populate_sdk_ext.bbclass', lineno: 66, function: copy_buildsystem
     0062:    import glob
     0063:    import oe.copy_buildsystem
     0064:    import subprocess
     0065:
 *** 0066:    subprocess.check_call("foo")
     0067:
     0068:    oe_init_env_script = d.getVar('OE_INIT_ENV_SCRIPT', True)
     0069:
     0070:    conf_bbpath = ''
File: '/usr/lib/python2.7/subprocess.py', lineno: 535, function: check_call
     0531:    The arguments are the same as for the Popen constructor.  Example:
     0532:
     0533:    check_call(["ls", "-l"])
     0534:    """
 *** 0535:    retcode = call(*popenargs, **kwargs)
     0536:    if retcode:
     0537:        cmd = kwargs.get("args")
     0538:        if cmd is None:
     0539:            cmd = popenargs[0]

We can't inject into methodpool at parsing time, since there may be
_append or other override operations against the function before its
execution.

(Bitbake rev: fae153095d23157dd7e72c29f683f86149ee33a8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 15:47:11 +00:00
Richard Purdie aece748762 bitbake: build/data: Don't expand python functions before execution [API change]
Right now, if you have some python code like:

X = "a"

def somefunction(d):
    d.setVar("X", "b")
    d.setVar("Y", "${X}")

then any sane person would expect that Y = "b" at the end of the
function. This is not the case, Y = "a".

This is due to the python function being expanded before execution, the
executed code would read d.setVar("Y", "a"). This understandably
confuses people, it also makes it near impossible to write ${} in a
python function without unintended things happening.

I think there is general agreement we should fix this and standardise
on non-expansion of python functions. We already don't expand anonymous
python (mostly).

I've checked OE-Core with buildhistory before and after this change and
there were a small number of issues this exposed which I've sent
patches for.

I propose we default to not expanding python code and then deal with
any consequences from that if/as/where identified. This will improve
new user understanding and usability of the system, it also allows
several long standing weird expansion issues to be fixed.

(Bitbake rev: 8bf33a8e92c0e188fa392030025756196c96fcbb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10 15:47:11 +00:00
Richard Purdie 4628fe12e7 bitbake: lib/bb: Add expansion parameter to getVarFlag
This sets the scene for removing the default False for expansion from
getVarFlag. This would later allow True to become the expand default.

On the most part this is an automatic translation with:

sed -e 's:\(\.getVarFlag([^,()]*, [^,()]*\)):\1, False):g'  -i `grep -ril getVar *`

There should be no functional change from this patch.

(Bitbake rev: 7c3b99c6a716095af3ffce0b15110e91fb49c913)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-04 23:47:49 +00:00
Andre McCurdy 68600aee2d bitbake: build.py: minor shell_trap_code() formatting tweaks
Fix quoting of $BASH_COMMAND and avoid wrapping at 80 columns (the
script which follows is likely to contain some very long lines, so
line wrapping in bb_exit_handler() looks somewhat out of place).

(Bitbake rev: 8e12c8f8441a7c6a03e603c5789d6037945704c1)

Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-30 11:43:55 +00:00
Chen Qi 74fa824e6c bitbake: bitbake: clean up stamp-base related codes
The 'stamp-base' and 'stamp-base-clean' related codes are no longer useful,
clean them up.

[YOCTO #8468]

(Bitbake rev: 7b4c42b315d4a26dd8f2ceb874a94737bf9f183e)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:28:55 +00:00
Richard Purdie 290534d3ff bitbake: build/utils: Add BB_TASK_IONICE_LEVEL support
Similarly to BB_TASK_NICE_LEVEL, add BB_TASK_IONICE_LEVEL which allows the ioprio
of tasks to be adjusted. This is in response to various qemu runtime timeouts
which have been witnessed on the autobuilder, seemingly due to IO starvation (we
already use NICE_LEVEL to adjust tasks). This has a fairly urgent need to deal
with certain 'random' failures we're seeing on the autobuilders in testing.

The format of the data in the variable is BB_TASK_IONICE_LEVEL = "<class>.<prio>".

For <class>, 2 is best effort (the default), 1 is real time and 3 is idle. You'd
need superuser privileges to use realtime. The <prio> value is a default of 4,
and can be set between 0 and 7 with 7 being lowest priority and 0 the highest.
The user can set this freely with normal privileges

Note that in order for this to take effect, you need the cfq scheduler selected
for the backing block device.

We could use nice wrapper functions for ioprio from modules like psutil however
that would complicate bitbake dependencies. This version has some magic numbers
but works on the main 32 and 64 bit x86 build architectures and can easily be
extended if ever needed. When we move to python 3.x, we can likely replace this
with standard calls.

(Bitbake rev: b9471ad147b102c45d65f5ffd9521864df7ff9c1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-29 07:35:52 +00:00
Christopher Larson 383450c78a bitbake: build: delete tasks thoroughly
We want addtask to be able to bring back a deleted task, but we don't want its
previous dependencies to come back with it, so rather than marking a task as
deleted and then skipping tasks marked as such, actually delete the task and
its dependency information in deltask.

While we're in that part of the code, also fix a couple 'not foo in bar'
instances to 'foo not in bar', which is preferred in python.

(Bitbake rev: 94b3f3d6bdfbfa47f7eb3c3de64940a145b2ddd1)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-03 12:33:08 +01:00
Andre McCurdy 1b88b7426a bitbake: build: Exit scripts with result of last shell function
Since shell scripts run with 'set -e' the final exit at the end of the
script can only be returning 0. However, for correctness and to follow
the original intention of the 'cleanup' commands, let's fix the typo
and return the success of the last shell function rather than the
success of unhooking the exit trap handler.

(Bitbake rev: bef724057f1ea81571dd3ac5a9cf862f818434b5)

Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-07-27 23:29:16 +01:00
Paul Eggleton d3809b77c4 bitbake: lib/bb: provide mechanism to bypass UI log suppression
The recent change to connect through the shell logging functions had an
unexpected side-effect - bb.error() and bb.fatal() cause a flag to be
set internally such that BitBake's UI will not print the full task log
on failure; unfortunately we have in places within the OpenEmbedded
metadata called these shell logging functions under error situations
where we still want to see the full log (i.e., the message we're sending
doesn't include the full error). Thus, provide a mechanism to fatally
exit with an error but unset the flag, utilising the built-in python
logging functionality that allows extra values to be passed in the log
record.

(Bitbake rev: e561b997c55e8537d82aa1339adfff4505cc38b7)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-07-16 15:09:26 +01:00
Paul Eggleton 3b50312307 bitbake: lib/bb: set up infrastructure for shell message functions
Create a fifo under ${T} for each running task that can be used by the
task to send events back to BitBake. The purpose of this is to allow
OpenEmbedded's logging.bbclass (which provides shell function
equivalents for bb.warn(), bb.note() etc.) to have those functions
trigger the appropriate events within BitBake so that messages are shown
through the BitBake UI rather than just in the task log; to do that we
just call the python functions.

Part of the fix for [YOCTO #5275].

(Bitbake rev: bae330a1f8a59a912eca71177b3c6ba7c92a2f38)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-07-12 22:50:40 +01:00
Richard Purdie 69b6919341 bitbake: bitbake: Add explict getVar param for (non) expansion
Rather than just use d.getVar(X), use the more explict d.getVar(X, False)
since at some point in the future, having the default of expansion would
be nice. This is the first step towards that.

This patch was mostly made using the command:

sed -e 's:\(getVar([^,()]*\)\s*):\1, False):g' -i `grep -ril getVar *`

(Bitbake rev: 659ef95c9b8aced3c4ded81c48bcc0fbde4d429f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-06-23 11:57:53 +01:00
Ed Bartosh db409697db bitbake: bitbake: reset build mtime cache before the build
Introduced build mtime cache structure. Reset it before the build
to prevent bitbake from crashing when build/tmp/stamps hierarchy
is removed.

[YOCTO: #7562]

(Bitbake rev: f8590547a198a78334debdf14bf40acb50c22ecc)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-04-29 10:59:54 +01:00
Richard Purdie 669c07d602 bitbake: build/data: Write out more complete python run files
Currently the output in the python task/function run files is rather
incomplete and effectively useless. This enhances the code to take
advantage of the bitbake's dependency tracking and extend the output to
include dependencies. This makes the files more usable for debugging
purposes. Since this only happens at python function execution time, the
overhead is minimal in the grand scheme of things.

(Bitbake rev: 02667e048c3e632f857c87177c0022eaf5481802)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-08-28 15:12:45 +01:00
Robert P. J. Day 6d08e5bb09 bitbake: lib/bb/*.py: Typo fixes/grammar/comment fixes, nothing functional.
(Bitbake rev: 587b144ee409d444494d8d7f2d1c53ede8f7c953)

Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-08-25 08:57:45 +01:00
Richard Purdie f3d08464ef bitbake: build.py: Ensure shared work siginfo files are written to the correct location
Right now shared work signature data is saved to the non-shared directory
which is confusing to everyone including bitbake. Whilst its messy,
extra the stampbase data instead, which ensures the sig data is written
to the correct location alongside its corresponding stamp file.

(Bitbake rev: 7ae1d4844d9d3a76f86ef32c5a794e51e334e588)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-06-19 17:55:11 +01:00
Richard Purdie 8a43a6a32b bitbake: build/utils: Fix broken exception handling
Checking for explicit exception names is bad, we also want to be able top
rely on inheritance. Fix these checks to be part of the real except clauses
so SkipPackage is recognised as being inherited from SkipRecipe.

(Bitbake rev: b131229145e1f2c372d6230a7b554e436c13c3f9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-06-01 14:29:32 +01:00
Richard Purdie 91a164cb87 bitbake: event: Add SkipRecipe event to replace SkipPackage
In the depths of time we were rather confused about naming. bb files
are recipes, the event to skip parsing them should be SkipRecipe,
not SkipPackage. This changes bitbake to use the better name but
leaves the other around for now. We can therefore start removing
references to it from the metadata.

(Bitbake rev: 98d9e6e0f514a7cb7da1d99bf4bd5602b89426d6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-06-01 14:29:31 +01:00
Alexandru DAMIAN f191ab0198 bitbake: toaster: measure task duration with server-side timestamps
The buildstats and toaster use separate time markers to measure the
duration of task execution. This causes a mismatch in the time
measured by buildstats class and the time measured in toaster.

The solution implemented here is to timestamp the creation of
every TaskBase event on the bitbake server side and calculate
the execution duration as the difference between creation time
of TaskSucceeded and TaskStarted events.

Based on an original patch by Marius Avram.

[YOCTO #5485]

(Bitbake rev: 7a08282c074c264f414cf7665dc873f51245072c)

Signed-off-by: Marius Avram <marius.avram@intel.com>
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-03-21 14:47:52 +00:00
Stefan Stanacar 510560338d bitbake: build: filter out expanded empty strings for lockfiles flag
If we have something similar to:

    LOCKFILES = "${TMPDIR}/my.lock"
    LOCKFILES_qemuall = ""

    do_task[lockfiles] += "${LOCKFILES}"

when expanded, lockfiles will be empty for qemu, resulting in

     File "/home/stefans/yocto/poky/bitbake/lib/bb/utils.py", line 630, in mkdirhier
	raise e
    OSError: [Errno 2] No such file or directory: ''

This should filter out the empty expansions.

(Bitbake rev: 7813e1bfd08cd48871f8c03cae2810265590105d)

Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-02-28 14:48:40 +00:00
Paul Eggleton 0008cd6444 bitbake: build: fix handling of task override for tasks with underscores in their names
Tasks whose names contain underscores (such as do_populate_sdk in OE)
when converted to a task override do not function properly. If we
replace underscores with hyphens we can still have a working override
for these tasks.

(Bitbake rev: cf90bd6b2a0ab7dce922bffb500d6a2ff2ff10e2)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-02-17 15:34:45 +00:00
Laurentiu Palcu 095bb006c3 bitbake: bitbake: build.py: create separate function for shell trap creation code
Currently, the shell trap code was created in exec_func_shell(). Split
the function so that we can create the code separately.

Also, some whitespaces were automatically deleted by my editor. Since
this is not necessarily a bad thing, leave these changes too.

(Bitbake rev: c712e622d20c61a07c9c172b60e9dc6beae14197)

Signed-off-by: Laurentiu Palcu <laurentiu.palcu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-02-09 10:01:38 +00:00
Robert Yang 1d3a139016 bitbake: bitbake/lib/bb/build.py: fix the task flags cleandirs
The user manual said:

'cleandirs' - directories which should created before the task runs but should be empty

But it only removes the dir, doesn't create it

[YOCTO #5703]

(Bitbake rev: 0636797d75874ce4577f29011d69c56a4c6b9e89)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-01-07 13:42:55 +00:00
Richard Purdie ba82035412 bitbake: build/ast: Create strong task add/del API in bb.build
Currently its near impossible to control task addition/deletion from
metadata context. This adds stong add/deltask API to bb.build
which is traditionally where it resided. The rather broken
remove_tasks function was removed, it didn't appear to do anything
useful or have any users.

This allows us to clean up hacks currently in use in metadata and use
standard API for it instead.

(Bitbake rev: bf7138dd38fc1f8efca80891198e3422fef64093)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-12-18 17:08:07 +00:00
Richard Purdie 8a82a3835c bitbake: ast/BBHandler/build: Add support for removing tasks (deltask)
Back in the depths of time we did support task removal. In the pre
AST days it was nearly impossible to continue supporting it, it wasn't
used so it was dropped. With the modern codebase we can easily now support
deltask and it would be very useful within the metadata since it can
massively simplify dependency trees.

As an example, a core-image-sato had 47703 inter task dependencies before
this patch and a patch to native.bbclass, afterwards with the noexec tasks
deleted, we had 29883. Such a significant simplification is worthwhile
and justifies adding a deltask operation to the system.

(Bitbake rev: acecbde6fb70ff3c96deab3cdf819d8442e87ed4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-12-18 17:08:07 +00:00
Peter Seebach 10717e5a0a bitbake: build.py: add single-quotes around already-expanded directory name
If the computed name of a directory contains an undefined variable
reference, bitbake dutifully creates a directory with a name that has
${...} in it. However, the actual task script created then tries to cd
to that directory, and the cd command fails, because no such directory
exists -- because the shell has helpfully removed the ${...} which did
not match any actual variables.

Since we want the name to be used exactly-as-is, add single quotes around
the name so this doesn't cause strange failures running tasks, which
allows us to progress past such failures and get to a point where they
can be diagnosed.

(Bitbake rev: 2809c2e6f2f35f9b08058950be896947ab5a0284)

Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-11-20 14:05:52 +00:00
Alexandru DAMIAN 85b9194171 bitbake: build, toaster: record proper task type
Bitbake tasks may be of type 'python' or 'shell',
or they may not be executed at all, which is record
as task type 'noexec'.

In order to record proper task type, this patch:

* creates no exec task type as the default value in
the toaster model definition

* adds full task flags to the bb.build.TaskStarted event
in build.py

* if the task actually starts, the toaster ui will
record the type of the task as either 'python' or 'shell'
based on the task flags.

[YOCTO #5073]
[YOCTO #5075]
[YOCTO #5327]

(Bitbake rev: 6648c57e6d369fc009ea3a9fe939def5d2c67bf5)

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-11-04 09:36:22 +00:00
Richard Purdie 35a1d37ac8 bitbake: build: Add BB_TASK_NICE_LEVEL to task code
On Linux its not possible for processes to regain a previous nice level after
it has changed. Its therefore not possible to have a core low priority and
then raise the priorities of individual tasks.

This variable allows us to do something like:

BB_TASK_NICE_LEVEL = "5"
BB_TASK_NICE_LEVEL_task-testimage = "0"

to give priority to specific tasks which the BB_NICE_LEVEL functionality
doesn't give us the option of.

(Bitbake rev: 94d82997220c6cfc7028f76719df028ba8254a5c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-09-22 12:19:44 +01:00
Alexandru DAMIAN 5ee82d4048 bitbake: bitbake: build, runqueue: adds info to the *runQueue* events
This patch adds task identifying information for all
runQueue and sceneQueue events, and for bb.build.Task* events.
This will allow matching event to specific tasks in the UI
handlers processing these events.

Adds RunQueueData functions to get the task name and task
file for usage with the runQueue* events.

Adds taskfile and taskname properties to bb.build.TaskBase.

Adds taskfile and taskname properties to the *runQueue* events

(Bitbake rev: b4a5e4be50d871a80dbe0993117d73f5ad82e38f)

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-09-22 12:19:44 +01:00
Richard Purdie 1181e69119 bitbake: build: Add logfile to add TaskBase events
We add the path to the logfile for all Task events except TaskInvalid
so that we can trace back the logfile locations at some future point.

TaskInvalid doesn't ever have a logfile.

(Bitbake rev: 8344d84c609446f59f9619cc7ca0d693b7e2bbd6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-09-16 13:12:34 +01:00
Richard Purdie 6e15fee9ee bitbake: build: Fix profile file names
Using the basename of the .bb file is not unique, for example xxx-native
and xxx can overwrite each other. If this happens whilst running, you can
get odd backtraces as one file is parsed as another tries to write out
new data.

Avoid issues by using PN for the output filename instead.

(Bitbake rev: c9534f8e59d44b885334607ed90a3be2e492ec69)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-09-01 15:51:11 +01:00
Nicolas Dechesne 9f0f799c46 bitbake: build.py: create symlink for run.do_xxx scripts
The 'courtesy' symlink for log.do_xxx are quite useful when debugging, so
with this commit, we now get similar 'courtesy' symlink for run.do_xxx
scripts.

We only create symlink for tasks, not individual functions.

The symlink is create right before the actual runfile is created, indeed
we cannot create the symlink right after running the task since a failure
or execption can happen, in which case the symlink wouldn't be created,
and symlink are particularely useful when the task failed!

Another option would be create the symlink after the runfile is created,
and before the script is executed, but that means we need to duplicate the
code in case of Shell vs Python task.

(Bitbake rev: a672b39c5d529ba85d72eee8fef4c4273eaa5397)

Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-08-23 17:29:54 +01:00
Peter Seebach f6ab9274bc bitbake: build.py: Provide useful diagnostics when exiting.
Running scripts with 'set -e' produces silent failures with no
diagnostic. Add an exit handler which produces diagnostics, including
details of what was running if the shell seems to be bash.

(Bitbake rev: e213e6a4c297a4f1c22eed15bd7b4cbc0e9eab4f)

Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-08-20 22:43:42 +01:00
Richard Purdie db6bd113d0 bitbake: build.py: Cleanup data store references
Clean up a number of old style accesses to the datastore.

(Bitbake rev: d872fef2c38749c3c6f5d84344db3ec2f9f134ce)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-06-20 15:10:32 +01:00
Richard Purdie 5af68d42ba bitbake: runqueue/build: Add recideptask flag
Currently, tasks like fetchall are slightly broken since if a recipe
has specific [depends] which occur after do_fetch and add items not listed
in DEPENDS and RDEPENDS, they are not caught by recrdeptask. We've gone
around in circles on this issue (e.g
http://git.yoctoproject.org/cgit.cgi/poky/commit/bitbake/lib/bb/runqueue.py?id=5fa6036d49ed7befe6ad50ec95c61a50aec48195
) and in many cases the behaviour of recrdepends is correct but tasks like
fetchall need the other behaviour.

To address this we add a recideptask flag which can be used in conjuction
with the recrdeptask flag to specify which task to to the inspection upon.
This means entries like do_rootfs[depends] which have do_fetch tasks are
caught and run.

I'm not 100% happy with needing another flag but I don't see any rational
way to get the correct behaviour in all cases without it.

[YOCTO #4597]

(Bitbake rev: f8c9b292b02ce2c28741b74901205f5e5807ca87)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-06-20 13:10:47 +01:00
Richard Purdie 3b4b782e07 bitbake: build: Avoid empty promises
The "see xxx for further information" is misleading since it is just
the same information. Clarify just to mention the that this is the location
of the logfile without any empty promise.

[YOCTO #4343]

(Bitbake rev: 7088c0e8553dd3c408b5bc06f8c34d5b72e9ea9a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-05-24 11:30:35 +01:00
Richard Purdie 4a081b5a52 bitbake: lib: Clean up various file access syntax
Python 3 is stricter about how files are accessed. Specficially:

 * Use open(), not file()
 * Use binary mode for binary files (when checksumming)
 * Use with statements to ensure files get closed
 * Add missing file close statements

(Bitbake rev: 9f08b901375ba640f47596f1bcf43f98a931550f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-05-09 22:28:04 +01:00
Paul Eggleton 406cb9920f bitbake: build.py: avoid deleting taint files when writing stamps
The stamp cleaning process that occurs before writing out new stamps for
a task was deleting taint files as well. This resulted in tasks that
were forcibly re-executed using the -f or -C command line options to
have their previous output restored from shared state when called upon a
second time, because the taint value was no longer incorporated into the
task signature and thus it was reverting to its previous value. This
also affected the kernel menuconfig command in OE-Core.

Note that the taint file *is* still deleted when doing -c clean, which
is the desired behaviour.

Fixes [YOCTO #3919].

(Bitbake rev: e6db0ee31178d4386802e720d75303ec7dc21519)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-02-22 05:22:39 -08:00
Richard Purdie cfb082961a bitbake: build.py: Dump out performance data of individual tasks
(Bitbake rev: 32aa49519e4f015e3c21466a7e5dc939f6369851)

Signed-off-by: Richard  Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-01-28 14:49:05 +00:00
Richard Purdie 39eb22bbc4 bitbake: build.py: Fix traceback when there are no dependencies
A recipe with no dependencies results in a traceback (e.g. all in
ASSUME_PROVIDED). This shouldn't happen and this patch fixes it.

(Bitbake rev: dee7decf87dfa8cb966fe40846d27f3e6ab1846b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-01-07 17:38:23 +00:00
Richard Purdie 525cc08cc5 bitbake: build.py: Preserve sigdata files in the stamps directory
Leaving the sigdata files around can aid debugging and doesn't harm
anything. This is the easiest way to allow this to happen.

(Bitbake rev: 1f500149ecd533a6edbeea902c3f1e009c755154)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-12-03 15:13:17 +00:00
Richard Purdie 9de050111b bitbake: build/siggen.py: Avoid removing too many stamps when cleaning
The "*" part of the mask is to ensure we clean both any stamp, and any
setscene varient. It turns out we would also trample other tasks,
e.g. do_package_write could trample do_package_write_rpm. do_package also
tramples do_package_write_* but this is less of an issue since the other
tasks depend on it.

Rather than use the wildcard, we can just use a list instead.

[YOCTO #3484]

(Bitbake rev: c14d831ea3f625e9a47266a0c4e6deefc924ca5a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-11-28 14:14:07 +00:00
Richard Purdie d485d0b0e8 bitbake: build/siggen: Add support for stamp 'clean' masks
Currently when we execute a task, we don't remove other potentially stale
stamps. This can mean if you switch between two different versions of a
recipe without a clean, the build can get very confused.

This patch adds in functionality to allow a wildcard expression of stamp
files to be removed when creating a new stamp file. This patch adds in
the core of the code to enable this but it also requires metadata support
to enable it.

When writing this improvement I went through several different options but
this was the only way I could find to allow things like noexec tasks to
function correctly (where stamps need to be created without the data store).

[YOCTO #2961]

(Bitbake rev: e026469b307522e5b6a680e0ae5587749d33dcae)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-09-20 11:48:38 +01:00
Richard Purdie 96ece9ad1a bitbake: build.py: Correct package to recipe in TaskBase events to use consistent terminology
(Bitbake rev: e5045429bce15b66c4355be214db3982ac7761f3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-08-18 16:21:45 +01:00
Andrei Gherzan af847d3637 bitbake: bitbake: build.py: Add stampdir argument to cached_mtime_noerror
After commit 2718537b4b (bitbake:
build.py: Only execute mkdirhier if stampdir doesn't exist) build
failes as cached_mtime_noerror needs an argument - stamp dir. This
argument was forgotten.

(Bitbake rev: e69103c4f7e45c24f1fbe9df0383f39e877abcb4)

Signed-off-by: Andrei Gherzan <andrei.gherzan@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-08-16 12:27:41 +01:00