Commit Graph

170 Commits

Author SHA1 Message Date
Richard Purdie 1181e69119 bitbake: build: Add logfile to add TaskBase events
We add the path to the logfile for all Task events except TaskInvalid
so that we can trace back the logfile locations at some future point.

TaskInvalid doesn't ever have a logfile.

(Bitbake rev: 8344d84c609446f59f9619cc7ca0d693b7e2bbd6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-09-16 13:12:34 +01:00
Richard Purdie 6e15fee9ee bitbake: build: Fix profile file names
Using the basename of the .bb file is not unique, for example xxx-native
and xxx can overwrite each other. If this happens whilst running, you can
get odd backtraces as one file is parsed as another tries to write out
new data.

Avoid issues by using PN for the output filename instead.

(Bitbake rev: c9534f8e59d44b885334607ed90a3be2e492ec69)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-09-01 15:51:11 +01:00
Nicolas Dechesne 9f0f799c46 bitbake: build.py: create symlink for run.do_xxx scripts
The 'courtesy' symlink for log.do_xxx are quite useful when debugging, so
with this commit, we now get similar 'courtesy' symlink for run.do_xxx
scripts.

We only create symlink for tasks, not individual functions.

The symlink is create right before the actual runfile is created, indeed
we cannot create the symlink right after running the task since a failure
or execption can happen, in which case the symlink wouldn't be created,
and symlink are particularely useful when the task failed!

Another option would be create the symlink after the runfile is created,
and before the script is executed, but that means we need to duplicate the
code in case of Shell vs Python task.

(Bitbake rev: a672b39c5d529ba85d72eee8fef4c4273eaa5397)

Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-08-23 17:29:54 +01:00
Peter Seebach f6ab9274bc bitbake: build.py: Provide useful diagnostics when exiting.
Running scripts with 'set -e' produces silent failures with no
diagnostic. Add an exit handler which produces diagnostics, including
details of what was running if the shell seems to be bash.

(Bitbake rev: e213e6a4c297a4f1c22eed15bd7b4cbc0e9eab4f)

Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-08-20 22:43:42 +01:00
Richard Purdie db6bd113d0 bitbake: build.py: Cleanup data store references
Clean up a number of old style accesses to the datastore.

(Bitbake rev: d872fef2c38749c3c6f5d84344db3ec2f9f134ce)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-06-20 15:10:32 +01:00
Richard Purdie 5af68d42ba bitbake: runqueue/build: Add recideptask flag
Currently, tasks like fetchall are slightly broken since if a recipe
has specific [depends] which occur after do_fetch and add items not listed
in DEPENDS and RDEPENDS, they are not caught by recrdeptask. We've gone
around in circles on this issue (e.g
http://git.yoctoproject.org/cgit.cgi/poky/commit/bitbake/lib/bb/runqueue.py?id=5fa6036d49ed7befe6ad50ec95c61a50aec48195
) and in many cases the behaviour of recrdepends is correct but tasks like
fetchall need the other behaviour.

To address this we add a recideptask flag which can be used in conjuction
with the recrdeptask flag to specify which task to to the inspection upon.
This means entries like do_rootfs[depends] which have do_fetch tasks are
caught and run.

I'm not 100% happy with needing another flag but I don't see any rational
way to get the correct behaviour in all cases without it.

[YOCTO #4597]

(Bitbake rev: f8c9b292b02ce2c28741b74901205f5e5807ca87)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-06-20 13:10:47 +01:00
Richard Purdie 3b4b782e07 bitbake: build: Avoid empty promises
The "see xxx for further information" is misleading since it is just
the same information. Clarify just to mention the that this is the location
of the logfile without any empty promise.

[YOCTO #4343]

(Bitbake rev: 7088c0e8553dd3c408b5bc06f8c34d5b72e9ea9a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-05-24 11:30:35 +01:00
Richard Purdie 4a081b5a52 bitbake: lib: Clean up various file access syntax
Python 3 is stricter about how files are accessed. Specficially:

 * Use open(), not file()
 * Use binary mode for binary files (when checksumming)
 * Use with statements to ensure files get closed
 * Add missing file close statements

(Bitbake rev: 9f08b901375ba640f47596f1bcf43f98a931550f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-05-09 22:28:04 +01:00
Paul Eggleton 406cb9920f bitbake: build.py: avoid deleting taint files when writing stamps
The stamp cleaning process that occurs before writing out new stamps for
a task was deleting taint files as well. This resulted in tasks that
were forcibly re-executed using the -f or -C command line options to
have their previous output restored from shared state when called upon a
second time, because the taint value was no longer incorporated into the
task signature and thus it was reverting to its previous value. This
also affected the kernel menuconfig command in OE-Core.

Note that the taint file *is* still deleted when doing -c clean, which
is the desired behaviour.

Fixes [YOCTO #3919].

(Bitbake rev: e6db0ee31178d4386802e720d75303ec7dc21519)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-02-22 05:22:39 -08:00
Richard Purdie cfb082961a bitbake: build.py: Dump out performance data of individual tasks
(Bitbake rev: 32aa49519e4f015e3c21466a7e5dc939f6369851)

Signed-off-by: Richard  Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-01-28 14:49:05 +00:00
Richard Purdie 39eb22bbc4 bitbake: build.py: Fix traceback when there are no dependencies
A recipe with no dependencies results in a traceback (e.g. all in
ASSUME_PROVIDED). This shouldn't happen and this patch fixes it.

(Bitbake rev: dee7decf87dfa8cb966fe40846d27f3e6ab1846b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-01-07 17:38:23 +00:00
Richard Purdie 525cc08cc5 bitbake: build.py: Preserve sigdata files in the stamps directory
Leaving the sigdata files around can aid debugging and doesn't harm
anything. This is the easiest way to allow this to happen.

(Bitbake rev: 1f500149ecd533a6edbeea902c3f1e009c755154)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-12-03 15:13:17 +00:00
Richard Purdie 9de050111b bitbake: build/siggen.py: Avoid removing too many stamps when cleaning
The "*" part of the mask is to ensure we clean both any stamp, and any
setscene varient. It turns out we would also trample other tasks,
e.g. do_package_write could trample do_package_write_rpm. do_package also
tramples do_package_write_* but this is less of an issue since the other
tasks depend on it.

Rather than use the wildcard, we can just use a list instead.

[YOCTO #3484]

(Bitbake rev: c14d831ea3f625e9a47266a0c4e6deefc924ca5a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-11-28 14:14:07 +00:00
Richard Purdie d485d0b0e8 bitbake: build/siggen: Add support for stamp 'clean' masks
Currently when we execute a task, we don't remove other potentially stale
stamps. This can mean if you switch between two different versions of a
recipe without a clean, the build can get very confused.

This patch adds in functionality to allow a wildcard expression of stamp
files to be removed when creating a new stamp file. This patch adds in
the core of the code to enable this but it also requires metadata support
to enable it.

When writing this improvement I went through several different options but
this was the only way I could find to allow things like noexec tasks to
function correctly (where stamps need to be created without the data store).

[YOCTO #2961]

(Bitbake rev: e026469b307522e5b6a680e0ae5587749d33dcae)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-09-20 11:48:38 +01:00
Richard Purdie 96ece9ad1a bitbake: build.py: Correct package to recipe in TaskBase events to use consistent terminology
(Bitbake rev: e5045429bce15b66c4355be214db3982ac7761f3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-08-18 16:21:45 +01:00
Andrei Gherzan af847d3637 bitbake: bitbake: build.py: Add stampdir argument to cached_mtime_noerror
After commit 2718537b4b (bitbake:
build.py: Only execute mkdirhier if stampdir doesn't exist) build
failes as cached_mtime_noerror needs an argument - stamp dir. This
argument was forgotten.

(Bitbake rev: e69103c4f7e45c24f1fbe9df0383f39e877abcb4)

Signed-off-by: Andrei Gherzan <andrei.gherzan@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-08-16 12:27:41 +01:00
Richard Purdie 2718537b4b bitbake: build.py: Only execute mkdirhier if stampdir doesn't exist
I noticed this was showing up on profile logs as a sigificant time user
in "bitbake bash" when bash was already built. It reduces the time from
5.2 to 4.5 seconds in my test environment.

We make use of the parser's mtime cache as once a directory exists,
we can assume it continues to exist and this avoids syscalls.

(Bitbake rev: 769b694eeb617bb793bd79d0d7b29c43d2646ece)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-08-16 11:19:02 +01:00
Richard Purdie a87aa7b7af bitbake: process: Improve _logged_communicate buffering
There are two problems with the _logged_communicate that are both
caused as a result of buffering I/O issues:

1) log truncation when python fails

2) While a bitbake task is running it is impossible to see what is
   going on if it is only writing a small incremental log that is
   smaller than the buffer, or you get only a partial log, up until
   the task exists.  It is worse in the case that stderr and stdout
   are separate file handles, because previous code blocks on the read
   of stdout and then stderr, serially.

The right approach is simply to use select() to determine if there is
data available and also flush the log before exiting.

This is based on a patch from Jason Wessel <jason.wessel@windriver.com>
with some changes to flush upon exit, abstract the non blocking file
descriptor setup and drop the buffer size parameter.

(Bitbake rev: 361fb71e907aa84c28ecec79fefc6ca39c39172f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-06-25 14:57:16 +01:00
Richard Purdie 15a31a93ba bitbake: taskdata.py: Add support for rdepends task flag
Currently its not possible to add arbitrary RDEPENDS to a specific task.
This can be useful and this patch adds functionality equivalent to the
'depends' task flag.

(Bitbake rev: db65080a6199baecc5c422294a4c4a9fd12dc29e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-06-25 14:57:16 +01:00
Paul Eggleton 5bd11a9bf3 bitbake: bitbake: ensure -f causes dependent tasks to be re-run
If -f is specified, force dependent tasks to be re-run next time. This
works by changing the force behaviour so that instead of deleting the
task's stamp, we write a "taint" file into the stamps directory, which
will alter the taskhash randomly and thus trigger the task to re-run
next time we evaluate whether or not that should be done as well as
influencing the taskhashes of any dependent tasks so that they are
similarly re-triggered. As a bonus because we write this file as
<stamp file name>.taskname.taint, the existing code which deletes the
stamp files in OE's do_clean will already handle removing it.

This means you can now do the following:

bitbake somepackage
[ change the source code in the package's WORKDIR ]
bitbake -c compile -f somepackage
bitbake somepackage

and the result will be that all of the tasks that depend on do_compile
(do_install, do_package, etc.) will be re-run in the last step.

Note that to operate in the manner described above you need full hashing
enabled (i.e. BB_SIGNATURE_HANDLER must be set to a signature handler
that inherits from BasicHash). If this is not the case, -f will just
delete the stamp for the specified task as it did before.

This fix is required for [YOCTO #2615] and [YOCTO #2256].

(Bitbake rev: f7b55a94226f9acd985f87946e26d01bd86a35bb)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-06-21 13:10:32 +01:00
Mark Hatle 799e076938 build.py: Add support for log and run filename changes
The format of the log file and run file are now selectable
using BB_LOGFMT and BB_RUNFMT, respectively.

The following values may be used:

{task} - task name
{taskfunc} - task.func or func, if task==func
{func} - function name, only available in BB_RUNFMT
{pid} - pid

The log/run files may be placed into a subdirectory that
is relative to T.

Default BB_LOGFMT is: log.{task}.{pid}
Default BB_RUNFMT is: run.{func}.{pid}

(Bitbake rev: 588da606eb81c52cb92d29041e1c67897427bfdf)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-23 18:15:24 +01:00
Mark Hatle 63aaecb057 build.py: Add additional debug messages
We now add a debug message when entering and exiting a python or
shell function.  This makes it easier to inspect a log and figure
out the run order from the logs.

(Bitbake rev: a0f554d0e722b6705844c6031fdcafa5d1a1c8a7)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-23 18:15:23 +01:00
Mark Hatle d0de551d44 build.py: Cleanup exec_func_shell
exec_func_python and exec_func_shell are similar, but variable
usage has diverged sync the two up.  Since exec_func_python is first
use that as the guide for the later exec_func_shell variable naming.

(Bitbake rev: ccfe1a3a2419172799957676107f240badf3f062)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-23 18:15:23 +01:00
Mark Hatle c9e38aed29 build.py: Add a log to capture task execution order
The new log.task_order contains an ordered list of the tasks as they
were executed in any given recipe.  The format of the lines is
<task> <pid>: <log file>

(Bitbake rev: 8662b43dcbd6e38a5b2ab9d6529af1fb08c146bf)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-23 18:15:23 +01:00
Christopher Larson 4d16bc3c5e bb.build: use relative log links, not absolute
Using symlinks to absolute paths isn't gaining us anything, and can cause
problems in situations such as automated build systems when they try to
resolve the link, depending on the mechanism used to pull the build artifacts
(e.g. nfs).

(Bitbake rev: f9cf2671c0ecad153db11bca1aebe151213bfb42)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-03-29 09:44:45 +01:00
Richard Purdie ff73b02a72 meta/classes: Convert to use appendVar and appendVarFlags
(From OE-Core rev: 3b57de68e70e77dbc03c0616a83a29a2e99e40b4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-03-05 10:36:53 -08:00
Paul Eggleton da5236cf07 bitbake/build: report TaskFailedSilent events as "Failed"
The change for setscene events to fire a TaskFailedSilent event instead
of TaskFailed resulted in "FailedSilent" being reported in the task
finish note log entry, which is not really desirable, so change it back
to reporting "Failed" again.

(Bitbake rev: 224bc74d4e901b7886b845fbb3b5fe7564a2f6cb)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-03-01 15:51:36 +00:00
Paul Eggleton 0e39107047 bitbake: implement BB_VERBOSE_LOGS
Enable configuring whether "set +x" is added to all shell tasks rather
than forcing it; this is enabled by setting BB_VERBOSE_LOGS to 1.

(Bitbake rev: 659411b6bb30e1a8355afc1c29b8170a8f2b55ac)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-02-28 13:47:10 +00:00
Paul Eggleton ce0ecf5d84 bitbake: fix setscene task start/failure handling
* When a setscene task starts, print out that it's starting in the UI
  (ensuring we get the correct task name)
* When a setscene task fails, ensure we remove it from the list of
  running tasks so that if you break out any time afterwards it
  is not still listed.

(Bitbake rev: e8a3499c95a6d4f2b8fed002fb9504733c5be3c6)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-02-28 12:27:22 +00:00
Dongxiao Xu 656f9a0758 Hob: A new implemetation (v2)
This commit implements a new design for hob
Some of the new features:
 - Friendly new designed GUI. Quick response to user actions.
 - Two step builds support package generation and image generation.
 - Support running GUI seprarately from bitbake server.
 - Recipe/package selection and deselection.
 - Accurate customization for image contents and size.
 - Progress bars showing the parsing and build status.
 - Load/save user configurations from/into templates.

(Bitbake rev: 4dacd29f9c957d20f4583330b51e5420f9c3338d)

Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Shane Wang <shane.wang@intel.com>
Signed-off-by: Liming An <limingx.l.an@intel.com>
Signed-off-by: Fengxia Hua <fengxia.hua@intel.com>
Designed-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-02-24 18:04:27 +00:00
Paul Eggleton eb6291b36f bitbake/knotty: avoid printing full task log when error already printed
If a task has logged an ERROR then don't print the contents of the
task's log file in knotty (the default terminal UI).

As a side-effect we now also respect BBINCLUDELOGS in knotty; if it is
false we never print the log (but the pointer to the log file is always
printed).

(Bitbake rev: b9746b7e4d7aa5c34eba15a61427bfc6949af123)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-01-15 10:06:22 +00:00
Paul Eggleton dcf6b641fe bitbake/fetch2: improve error formatting for fetcher errors
* The "name" argument to FuncFailed is rarely used as a name in actual
  usage within bitbake, so don't treat it as one in the output.
* Don't print URL for FetchError if it was not specified (i.e. don't
  output "Fetcher failure for URL 'None'")
* Don't include URL in "unable to fetch from any source" message since
  we supply it to FetchError and it will be printed anyway.
* Don't include URL in "checksum failed" message for the same reason

(Bitbake rev: 86811bd85e2e453ee92a05fe60160d9b49ac69e8)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-01-15 10:06:21 +00:00
Richard Purdie aab6939cb1 build.py: Be determistic about a function's cwd
There is a subtle but nasty problem that a function's cwd can vary
depending on whether ${B} (often ${S}) exists before the funciton is
called or not. Most functions in the system can cope with this but
its bad practise and I've just witnessed build failures resulting
from this during image generation from bootimg.bbclass. I also
suspect this could explain some odd fetcher behaviour witnessed in
the past.

This change ensures we always call funcitons with a specific build
directory making things deterministic.

(Bitbake rev: ef0888f83fa4408eb768257d7e03700202faad18)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-12-01 12:06:27 +00:00
Richard Purdie 4cd9671078 bitbake: Update users of getVar/setVar to use the data store functions directly
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-11-27 10:35:30 +00:00
Paul Eggleton dea2fd181a bitbake/lib/bb/msg.py: fix setting debug and verbosity levels
The debug and verbosity levels (as set by the -D and -v command line
options respectively)  were not being passed through within msg.py since
bitbake revision 45aad2f9647df14bcfa5e755b57e1ddab377939a due to
incorrect variable names.

Fixes [YOCTO #1513].

(Bitbake rev: c6e88b7c0e61f9586a275df53f48b90687c5f92f)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-09-25 16:25:27 +01:00
Richard Purdie e7981ad10d build.py: Fix cut and paste error
(Bitbake rev: a1980fc89e45be6fbd4a193aaa8142e8bebcf3f4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-08-15 12:40:45 +01:00
Richard Purdie 7ee93b206a bitbake/logging: Overhaul internal logging process
At the moment it bugs me a lot that we only have one effective logging
level for bitbake, despite the logging module having provision to do
more advanced things. This patch:

* Changes the core log level to the lowest level we have messages of
  (DEBUG-2) so messages always flow through the core logger
* Allows build.py's task logging code to log all the output regardless
  of what output is on the console and sets this so log files now
  always contain debug level messages even if these don't appear
  on the console
* Moves the verbose/debug/debug-domains code to be a UI side setting
* Adds a filter to the UI to only print the user requested output.

The result is more complete logfiles on disk but the usual output to the
console.

There are some behaviour changes intentionally made by this patch:

a) the -v option now controls whether output is tee'd to the console.

Ultimately, we likely want to output a message to the user about where the
log file is and avoid placing output directly onto the console for every
executing task.

b) The functions get_debug_levels, the debug_levels variable, the
set_debug_levels, the set_verbosity and set_debug_domains functions are
removed from bb.msg.

c) The "logging" init function changes format.

d) All messages get fired to all handlers all the time leading to an
increase in inter-process traffic. This could likely be hacked around
short term with a function for a UI to only request events greater than
level X. Longer term, having masks for event handlers would be better.

e) logger.getEffectiveLevel() is no longer a reliable guide to what
will/won't get logged so for now we look at the default log levels instead.

[YOCTO #304]

(Bitbake rev: 45aad2f9647df14bcfa5e755b57e1ddab377939a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-08-15 09:14:28 +01:00
Robert Yang 671e580111 bitbake: Add task specific stamp file support
This patch, based on proof of concept code from Richard adds code to
bitbake to allow individual tasks to optionally specify their stamp
file using the stamp-base flag. This takes the same form as the STAMP
variable but can be specified on a per task basis.

Code is also added to runqueue to ensure that if two tasks share the
same stamp file, only one will be executed at once.

A significant usecase for this code is to share source code (${S})
between recipes where separate build directories (${B}) are used.

(Bitbake rev: 41bef02bef8379590ba012319aebe05068a8081e)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-06-28 13:44:38 +01:00
Mark Hatle a1f79a7896 runqueue.py: Add umask task control
The umask for a task can now be set as:

task[umask] = 022
task[umask] = '022'

If specified as a text string, it must be octal.  (This is due to
recipe parsing where it's always set to a string.)

[RP tweaked to use None instead of -1]
(Bitbake rev: a5caaaaba8f0db1af5d8f2e610021d6d4b56894e)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-06-24 12:23:12 +01:00
Richard Purdie bdab8e9354 bitbake/build.py: Drop exec_shell environment as its now unneeded
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-06-08 20:38:28 +01:00
Richard Purdie 355338c40c bitbake/cooker.py: Misc sync with upstream
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-06-08 20:38:27 +01:00
Richard Purdie fe5f742865 build/siggen: Ensure a task signature file is generated for each task that is executed
Tracing down signature changes and dependencies is hard unless the complete task
chain can be evaultated. Since we have the data available at task exeuction
time, writing it to disk makes sense.

This change is suboptimal internal knowledge stamp_internal() has leaks into
make_stamp() and the concepts used by this code need revisiting but the change
brings enough usability benefit to warrant its inclusion despite that.

Addresses issue [YOCTO #1074]

(Bitbake rev: 139b8a625818225c358a1b8363518d7ed6913188)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-05-25 23:39:37 +01:00
Chris Larson 2d73cd1d32 build: fix dir removal traceback
This one is to cover the case where the current directory vanishes out from
under us, so os.getcwd() raises an OSError.

(Bitbake rev: 7a29ab534388c0095f7f826b16c5cff343927d10)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-05-06 15:46:03 +01:00
Chris Larson 8b938dc82c build: in exec_func, mkdirhier ${T}
This should fix the -c clean traceback people are seeing.

(Bitbake rev: 416d24912fcef1d82ce2c02855accd86a29e76b2)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-05-06 15:42:54 +01:00
Chris Larson 88dbb0523c Initial work on getting bitbake working under pypy
- use os.chmod, not os.fchmod, as the latter is missing under pypy
- rearrange our imports a bit
- don't die if sqlite3 is missing shared cache support

(Bitbake rev: f229824dc9c453adf6067500e2bf6761536e4f2f)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-05-06 15:42:03 +01:00
Richard Purdie e41671ea1d Revert "bitbake/gcc: Enable a shared common source tree"
This reverts commit 12b163dbd8 as it
was an unintended change for master
2011-03-29 00:34:11 +01:00
Richard Purdie 12b163dbd8 bitbake/gcc: Enable a shared common source tree
This patch is a quick proof of concept to show how source code could
be shared between recipes which use ${B} to have a separate build
directory compared to source directory ${S}.

Issues:

a) gcc uses sed and creates config files against ${S} which means
the directory should not be shared. Need to change the way that works.

b) Could be extended to cover eglibc except there is a patch applied
against nativesdk versions which again makes the source incompatible.

c) Need to clean up the layout in work-shared and make a directory level deeper
to ensure patch separation.

d) clean task does not remove stamps

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-03-28 15:09:50 +01:00
Chris Larson c07a153e72 build: add missing newline
(Bitbake rev: a7aa0415bdaa458a941004bf8dd8bbfeddd6ef5a)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-03-03 22:51:28 +00:00
Chris Larson 043b70419e build: switch to old cwd handling
We want this to ensure the user can run the run. script from anywhere.

(Bitbake rev: a600b79ecefc95eeb266c3f362c7160fa8c948c1)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-03-03 22:48:39 +00:00
Richard Purdie 91c0b3a2e6 bitbake/build.py: Fix TaskFailed parameters
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-02-23 15:06:34 +00:00
Richard Purdie fb9c5a196e bitbake: event/build: Drop stdout specific handling for python messages as this is no longer needed with newer log handling
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-02-07 14:50:27 +00:00
Richard Purdie c34913ea5a bitbake/build.py: When looking up extra stamp info for setscene tasks, use the real taskname
(Fixed based on a bug reported by Dongxiao Xu <dongxiao.xu@intel.com>)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-01-27 11:20:58 +00:00
Dongxiao Xu 80cf3e405c bitbake: Introduce stamp-extra-info task flag into stamp filenames
For certain tasks, we need additional information in build stamp file
other than the task name and file name. stamp-extra-info is introduced as
a task flag which is appended to the stamp file name.

[Code simplifcations/tweaks from Richard]

Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-01-18 12:50:04 +00:00
Richard Purdie 72b452ecf2 bitbake/build.py: Allow FuncFailed error to have an optional message
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-01-17 23:54:28 +00:00
Richard Purdie 0388f0758f bitbake/build.py: Rename message field to avoid DeprecationWarning: BaseException.message warning
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-01-17 15:52:06 +00:00
Bernhard Reutner-Fischer bff1d2ba06 build: remove duplicate import of utils
(Bitbake rev: 86c6434f093ce5e55d1febfb424a8e595ffc2d17)

Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-01-12 13:47:18 +00:00
Bernhard Reutner-Fischer 2ef204dcdd *: use utils.remove() some more
(Bitbake rev: d3489b141cac1197324661680fe38b8a88bc49b4)

Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-01-12 13:47:14 +00:00
Richard Purdie 5da9747024 bitbake/build.py: Fix del_stamp work correctly after recent stamp function changes
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-01-07 14:03:27 +00:00
Richard Purdie 66c6200ff3 bitbake build/siggen/runqueue: Fix stampfile parameters
The current parameters are not useful to the stampfile generator function
as they can't uniquely define a task. This updated things so the
parameters can identify unique tasks.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-01-06 19:47:03 +00:00
Richard Purdie ec6e4793dc bitbake build.py: Stamp handling improvements
* Move stamp file deletion out of the internal stamp helper function
* Add a new function to return the path to a stamp for a given task

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-01-06 19:43:56 +00:00
Richard Purdie 4a10a5b438 bitbake build.py: Use localdata for stamp handling, not d
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-01-06 19:40:41 +00:00
Richard Purdie 51a2e8712a bitbake/build.py: Ditch the extract_stamp function as its just going to complicate improving the stamp handling functions
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-01-06 15:41:39 +00:00
Chris Larson e7febaa343 build: use logger.isEnabledFor, not getEffectiveLevel
I think this is less confusing, and avoids needing to know about the *range*
of logging levels, instead simply asking what we really want to know.

(Bitbake rev: dc2264387617586b5c0a61e126c75edde5e7abcd)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-01-06 10:46:54 +00:00
Richard Purdie 6ded4e0ae7 bitbake: build.py: Logging of None was losing the logs so force this to stdout so we can see function execution failures
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-01-05 18:57:07 +00:00
Richard Purdie 0f70631544 bitbake: Misc syncups with bitbake upstream, mostly whitespace and a missed line from the last UI commit
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-05 11:24:14 +00:00
Richard Purdie 25ac24e02e bitbake: build.py logging updates
python tasks calling shell functions using exec_func() would show the log
file as /dev/null. It makes most sense for all the task logging to be setup
centrally by exec_task(), at least with the current code base in Poky.

This commit will need discussion in relation to upstream bitbake and the
IO redirection could be better handled using a context manager (although
task contexts shouldn't ever nest).

Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-05 00:58:24 +00:00
Chris Larson 93e20c51c4 Use os.devnull, not /dev/null
(Bitbake rev: ce5ab0fc524a1c2c48c4c39d6fc8aae23019207b)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:52 +00:00
Chris Larson 7bbde5b149 build: fix FuncFailed for the no logfile case
(Bitbake rev: 3a4b5eb20e41bd3cecd9d7e8dff7d076d8a7bb9a)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:50 +00:00
Chris Larson 3d51fd2b7d build: ensure LogTee has a valid name property
(Bitbake rev: 0ebb46e25261cfc85aaef2790cba7c1ec057c306)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:50 +00:00
Chris Larson 6f80455777 build: fix -D with shell functions
(Bitbake rev: 1c8be64732fdf4f3a608c090b3dc92065d6058d6)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:48 +00:00
Chris Larson aaa55880ac build: send logging messages to the log file for python functions
(Bitbake rev: ee1cce6ab21ddda60a7a070d03e98ff8485a5e71)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:48 +00:00
Chris Larson e2363f3cdd build: use a contextmanager for locks
Also don't bother passing logfile to exec_func_python, at least until we start
adding the logfile as a file handler to the bitbake logger.

(Bitbake rev: f99ee4680c9f67b7ed13fc06044ba2382f9a782c)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:48 +00:00
Chris Larson b4eff9fcef build: use bb.process instead of os.system
(Bitbake rev: 53740977521bc81ffa37adfa7bbeb8f2a80ea165)

build: write logfiles per task, not per function
Based on d14f9bf6 from poky, reworked for master and other cleanup.

(Bitbake rev: beadff2eca1eb95f0411115dd72ddb4c3c44c604)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:48 +00:00
Chris Larson 666513982f Fix FuncFailed construction
(Bitbake rev: 085e66f9c14123ea2c0f1e34f7737cf77071f86d)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:36 +00:00
Chris Larson 136a631b2e Restore FuncFailed argument compatibility
(Bitbake rev: c07cc08f7fd503ac3013ccc43c79198c4c3b7b29)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:36 +00:00
Chris Larson 311c9f5042 Only reference logfiles that exist
(Bitbake rev: 760f647ba044009150ee219869fc9dea171a7535)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:36 +00:00
Chris Larson c90bfa57f5 Fix bitbake -k issue introduced by build exception cleanup
A SystemExit from a python function wasn't being raised as a FuncFailed, which
resulted in it not being caught by the exception handlers in the runqueue for
the worker process, which resulted in a SystemExit exit, rather than os._exit,
which causes all manner of problems when used in a forked process.  This fixes
it by ensuring we raise a FuncFailed when seeing exceptions which aren't
instances of Exception.

(Bitbake rev: dafe92fe9f387450d9f9e9ff41c99388998b7495)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:36 +00:00
Chris Larson 9897d56861 Simplify build exception handling
- Drop EventException
- Use FuncFailed as the primary function failure exception, using TaskFailed
  for the event (leaving it up to the process running exec_{func,task} to
  display the more detailed information available in the exception).
- Switch InvalidTask to an exception rather than an event, as that's a
  critical issue.
- Reduce the number of messages shown to the user when a task fails -- they
  don't need to be told it fails 12 times.  Work remains in this area though.

(Bitbake rev: 06b742aae2b8013cbb269cc30554cff89e3a5667)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:35 +00:00
Chris Larson ecc68fa4fb Switch bitbake internals to use logging directly rather than bb.msg
We use a custom Logger subclass for our loggers

This logger provides:
- 'debug' method which accepts a debug level
- 'plain' method which bypasses log formatting
- 'verbose' method which is more detail than info, but less than debug

(Bitbake rev: 3b2c1fe5ca56daebb24073a9dd45723d3efd2a8d)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:33 +00:00
Richard Purdie b80a8d28f8 bitbake: Add support for 'noexec' tasks
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-11-06 14:33:05 +00:00
Richard Purdie d14f9bf6c4 bitbake/build.py: Write logfiles on a per task basis, not per function
Per function logging made it hard to track down what was happening so
switch to logging on a per task basis.

[BUGID #383]

Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-10-11 12:53:54 +01:00
Mark Hatle 8e429accb8 build.py: Add a symlink for the logging
[BUGID #375]

When configuring for the log file output, we generate a symlink to the
target filename.  This link uses the same file naming, but without the
active pid, making it easier to see which log file is the last one
generated.

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
2010-10-01 18:45:32 +01:00
Richard Purdie 89c7260931 bitbake/build.py: Set BB_FILENAME to represent the .bb file being built (including any virtual prefix)
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-09-13 15:39:07 +01:00
Richard Purdie 332c33af18 bitbake/data.py: Add emit_func() and generate_dependencies() functions
These functions allow generation of dependency data between funcitons and
variables allowing moves to be made towards generating checksums and allowing
use of the dependency information in other parts of bitbake.

Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-08-31 12:47:41 +01:00
Richard Purdie 616e75523b bitbake: Correctly route events from the worker to the server
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-08-20 12:26:19 +01:00
Richard Purdie b6bfe14205 bitbake: Switch to use subprocess for forking tasks and FAKEROOTENV to run shell and python under a fakeroot environment
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-08-19 22:36:25 +01:00
Richard Purdie 9ef9b5e1a7 bitbake/build.py: Note when calling functions that don't exist. Should probably error
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-08-12 14:44:07 +01:00
Richard Purdie dfd7e1f76b bitbake/build.py: Set the current taskname in the BB_CURRENTTASK variable
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-08-12 14:41:31 +01:00
Bernhard Reutner-Fischer 8105d4381f *: Fix typo in documentation
s/dictonary/dictionary/

(Bitbake rev: 0cc632761e75f66a8ce5ca2fe370f7551ccbfdf0)

Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-08-04 16:12:39 +01:00
Richard Purdie 0f516fe9a0 bitbake/build.py: Add support for pre and postfuncs for tasks
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-08-04 11:43:12 +01:00
Chris Larson 6be4fa575a Switch a fatal usage to SystemExit ("T" not set)
This failure is one the *user* needs to see, as it may have to do with their
setup, but where they don't need to see a traceback and all, since they're not
developers, so I think SystemExit is appropriate here.

(Bitbake rev: e3cc9bc7b6ef1a2eca78235790a051329bceb9a4)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-07-02 15:41:36 +01:00
Chris Larson 562fd5f2a7 Drop unnecessarly duplicated messages wrt FuncFailed
(Bitbake rev: 9ec9c0e560d4fa6df74a90f598f9db92713525a2)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-07-02 15:41:36 +01:00
Chris Larson c3eae29efa Don't try to expand non-string values
(Bitbake rev: fe36a726b9f930bbd6fd758c0aee78559e95f02b)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-07-02 15:41:34 +01:00
Chris Larson 1180bab54e Apply some 2to3 transforms that don't cause issues in 2.6
(Bitbake rev: d39ab776e7ceaefc8361150151cf0892dcb70d9c)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-07-02 15:41:33 +01:00
Chris Larson bcd9c57a96 Switch some references to moved functions
(Bitbake rev: ddea2978cb969cf1381cfee5c055de0c9e56a4e3)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-07-02 15:41:33 +01:00
Chris Larson 1c74fd768f Import fixups
(Bitbake rev: 4fa052f426e3205ebace713eaa22deddc0420e8a)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-07-02 15:41:32 +01:00
Chris Larson 7acc132cac Formatting cleanups
(Bitbake rev: 2caf134b43a44dad30af4fbe33033b3c58deee57)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-07-02 15:41:32 +01:00
Chris Larson 94b60d1247 Consolidate the exec/eval bits, switch anonfunc to better_exec, etc
The methodpool, ${@} expansions, anonymous python functions, event handlers
now all run with the same global context, ensuring a consistent environment
for them.  Added a bb.utils.better_eval function which does an eval() with the
same globals as better_exec.

(Bitbake rev: 424d7e267b009cc19b8503eadab782736d9597d0)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-07-02 15:41:31 +01:00
Joshua Lock 43f0d63237 bitbake: allow fakeroot functionality to be provided by an alternative program
If the variable FAKEROOT is set its value will be used, otherwise we default
to fakeroot.

Signed-off-by: Joshua Lock <josh@linux.intel.com>
2010-06-25 14:54:51 +01:00
Richard Purdie fdf134abfb bitbake/build.py: No need for this gross hack now
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-01-27 12:40:02 +00:00