We need this if we want to run the buildWorldTargetList function more than
once, for example in a UI where we can change the MACHINE and DISTRO as much
as we like before triggering a build.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
When we're running a long operation with indeterminate duration it's useful
to use the gtk.ProgressBar's pulse method to show that something is happening
but we don't know how long it will take.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Provide a gtk.ListStore subclass which includes a function,
populate(), which takes as input the data emitted by
bb.event.TargetsTreeGenerated and fills the ListStore model
appropriately.
Furthermore convenience functions are provided by which the caller can
get gtk.TreeModel subclasses which provide filtered views of the data.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Add summary, license and group metadata to RecipeInfo and the cache.
Unfortunately this impacts parse speed but gives us a much richer set of
metadata to expose through UI's which can be accessed via the
generateTargetsTree command.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Move runqueua and taskdata initialisation into a new function,
prepareTreeData(), so that generateDepTreeData() and
generateTargetsTreeData() are not duplicating the same logic.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
The generateTargetsTree() command needs to return a model which includes more
metadata than the one generated by generateDepTree().
This patch adds a new method generateTargetsTreeData() to the cooker, based
on generateDepData(), and switches generateTargetsTree() to use it.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Some configuration variables (MACHINE, MACHINE-SDK and DISTRO) set which
confguration files bitbake should use.
The added command , findConfigFiles, enables a UI to query which files are
suitable values for a specified parameter.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Add a new command generateTargetsTree() which returns a dependency tree of
possible targets (tasks and recipes) as well as their dependency information.
Optional parameter 'klass' also ensures any recipes which inherit the
specified class path (i.e. 'classes/image.bbclass') are included in the model
Signed-off-by: Joshua Lock <josh@linux.intel.com>
In the none server, events don't get processed unless the idle_commands
function gets called, which previously wasn't happening with getEvent();
thus UIs that use this to get events were not working.
Fixes [BUGID #561]
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Remove some comments, imports etc. to do with XMLRPC (inherited from xmlrpc.py
which this file was based upon.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If expanding a variable triggers an exception the caller currently has no
way to supress the error message or otherwise handle the siutation. An
example of where this is a problem is "bitbake -e" showing tracebacks and
errors for variables like SRCPV in OE/Poky.
Secondly in a chained expansion fails, log mesages are recorded for
every step of the expansion, not just the innermost error which is
where the real failure occured.
To fix this we introduce a new exception ExpansionError which callers
can handle as appropriate.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if a variable has been set with ??= and the code looks it up
before the data finalisation phase, no value is found. This is causes
serious problems for anonymous python functions which manipulate data, or
for the fetcher revision handling code where revisions can be set with
??=.
There is also a significant performance implication for processing lazy
assignment in finalise.
Moving the check for a default value into getVarFlag addresses both
the timing issue and the performace. This change gives a 7% real time
performance improvement to parsing the Poky metadata. The cost of the
check at this point is minimal since we have all the data flags available.
This should also fix Yocto bug 752.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When parsing if a SystemExit event is triggered, it causes the parsing thread to
exit and the main process hangs waiting for it to finish indefintely. Add code to
catch BaseExceptions and raise these with the main process gracefully instead
of just hanging indefinitely with zombie processes.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This avoids cases where the stats are modified after the event is fired but
before it's dispatched to the UI.
(Bitbake rev: 1954f182687a0bd429175dda87f05d8a94bb403a)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake/lib/bb/process.py:15: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This create a clean() method in each of the fetcher modules
and correctly cleans the .done stamp file and lock files
Signed-off-by: Saul Wold <sgw@linux.intel.com>
This allows fetching git repositories using Kerberos authentication.
(Bitbake rev: d761cf98284b02eb3d3a1f879782c501c284b698)
Signed-off-by: Javier Martin <javier.martin@vista-silicon.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use '==' instead of 'is', otherwise it will always return
true since 'rev' and "SRCREVINACTION" are not the same object.
(Bitbake rev: f30b3af975a071d1584817054a2996f08a3aba4f)
Signed-off-by: Javier Martin <javier.martin@vista-silicon.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I forgot that bin/bitbake is what does the environment filtering based upon
BB_ENV_WHITELIST, etc.
(Bitbake rev: 2cc6b6951bd17832866ec710029d119d2df31ba4)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use comments instead of docstrings where appropriate in CacheData.
(Bitbake rev: 088d516e02bb2b4ce8a50bbaa967e944c46e620b)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This script has subcommands which operate against your bitbake layers, either
displaying useful information, or acting against them. Currently, it only
provides a show_appends command, which shows you what bbappends are in effect,
and warns you if you have appends which are not being utilized.
Currently, a bug exists when using this due to the DataContext stuff, but I'm
not certain as to the root cause, it appears to be the bb package relying
implicitly on the way the bitbake script does things. A fix for that issue
will be forthcoming, as will further subcommands.
(Bitbake rev: 78b6d4cb26cec3321f8eec9889205a6b93b2ee18)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Enable the fetcher to be able to unpack and SRPM. By default the system will
unpack the contents of the SRPM into the WORKDIR.
A new syntax "unpack=file" was developed for the SRC_URI, to allow for a
recipe to extract a specific file within an SRPM. An unpack operation will
then be executed on the extracted file.
In order to apply extracted patches (or unpack files not specified with
unpack), you must specify the path using WORKDIR, i.e.:
file://${WORKDIR}/mypatch.patch
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
See the problems in http://bugs.python.org/issue1692335, need to set self.args
correctly.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Drop some old md5 functions since we have improved functionality now which includes
sha256 checksum support. This stops each download being md5 checksumed twice.
Also change ".md5" stamp extentions to ".done" to better describe its use as a
download complete marker file and no longer write the md5 sum to the files.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We no longer need index/workdir support in the mirror tree and it causes all
kind of reference naming problems.Simplifying the code to remove this and use
just bare clones addresses this problem.
We increase the "version" number on the mirror tarballs to reflect the change
and ensure older mirror tarballs are not used as they would break.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since local mirror fetches are always symlinked from the download directory
directly, there is no need for this premirrors hack which doesn't cover
mirrors and also abuses the localpath variable with inconsistent results.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When files are fetched from a mirror source that happens to be local,
ensure links are created for the file since subsequent fetch calls
can then follow the links to find files.
Any other approach such as the existing manipulations of localpath
internally to the fetcher are prone to errors, races and other issues.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When processing a cvs SRC_URI to a file:// mirror, the user and host information
will break the mirror processing. This patch addresses it by only constructing
valid urls.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* SRC_URI format:
the SRC_URI are extended to allow multiple src rev:
name=<name1>,<name2>,...<name-n>
branch=<branch1>,<branch2>,...,<branch-n>
also SRCREV can be defined with
SRCREV_<name1> = xxxxx
SRCREV_<name2> = xxxxx
* FetchData extention
to support multiple src rev, several FetchData data are added:
- FetchData.names: list of name in SRC_URI, one name per srcrev. name is the index of revision and branch
- FetchData.revisions: dictionary of name->revision.
- FetchData.branches: dictionary of name->branch.
For example, linux-yocto recipes becomes:
SRC_URI = "git://git.pokylinux.org/linux-yocto-2.6.37;protocol=git;branch=${KBRANCH},meta;name=machine,meta"
FetchData.names = ['machine', 'meta']
FetchData.revisions = { 'machine':xxxxx, 'meta':xxxxxx }
FetchData.branches = { 'machine':${KBRANCH}, 'meta':'meta'}
* generic revision handling extension
the related revision handling code in fetch2.__init__.py are changed accordingly. the major change is add name parameter to indicate which src rev to handling. originally there is one src rev per FetchData, so FetchData parameter is enough. now since one FetchData has multiple src rev, it is necessary to use FetchData + name to specifiy src rev.
* git extension
git fetcher are also revised to take advantage of the multiple src rev in FetchData. especially the download() method are enhanced to fetch multiple src rev.
* other fetcher (svn, hg, ...) does not support multiple src rev. they just sync the API to add name, and then simply ignore the name. no actually functional change
Signed-off-by: Yu Ke <ke.yu@intel.com>
Since we're now always providing the git source control files it becomes
pointless to handle the tarballs of specific git revisions so drop this
part of the fetcher.
Signed-off-by: Yu Ke <ke.yu@intel.com>
The git download method clones the git repository to the local machine. The unpack process
can be optimised to be a local to local machine clone or a direct readtree operation to the
destination using git.will clone git repo to local, so git unpack can be simplified
to only checkouting the code to the work dir. For fullclone case, we also
need to manually copy all the ref info, which is needed by the later do_kernel_checkout().
Rather than use hardlinks, we reference the repository using alternatives since the
download directory may be on a different filesystem.
[Change to use -s by Richard Purdie]
Signed-off-by: Yu Ke <ke.yu@intel.com>
The server UI was reading 1024 bytes, then sleeping for 0.25 seconds. Since
most new LogRecord events are larger than this it leads to a build up of data
which is only processed slowly, leading to a bottleneck and a slow down of
all bitbake processes.
Thanks to Dongxiao Xu <dongxiao.xu@intel.com> for the great work in debugging
this. A large value has been left in for the read() command just to ensure some
fairness amongst process handling if a task tries to log truly huge amounts of
data to the server, or goes crazy and ensures the main loop doesn't stall.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sometime user want a purely local fetching, i.e. using local mirror without
any remote netowrk access. BB_NO_NETWORK option is introduced for this purpose
check_network_access() is the guard for BB_NO_NETWOKR option. it should be
put in any place that fetcher use network access
Signed-off-by: Yu Ke <ke.yu@intel.com>
the download is to fetch the source from URL, the build_mirror_data is
to create the mirror tar ball. the original go() method mix them together,
it is more clean to split them.
Signed-off-by: Yu Ke <ke.yu@intel.com>
there is case that we need to distingush bb.fetch and bb.fetch2,
and use different API for bb.fetch and bb.fetch2. so it is necessary
to add version info for distinguish purpose
Signed-off-by: Yu Ke <ke.yu@intel.com>
change the unpack to use the urldata and rootdir parameter
- urldata is the FetchData instance
- rootdir is the dir to put the extracted source. the original unpack
use current dir (os.getcwd) as destination dir, which is not flexible
and error-prone (error will occur if caller not chdir to dest dir)
Signed-off-by: Yu Ke <ke.yu@intel.com>
current bitbake-diffsigs simply print out the whole 'runtaskdeps' when there's mismatch, which
is not very readable. On the other hand, 'runtaskhashes' comparison is broken which assumes
same key existing in two sides. This commit provides better output by figuring out differences
from addition, removal or hash change.
Signed-off-by: Kevin Tian <kevin.tian@intel.com>
Take a real world testcase where you have two recipes, each of which
contains PACKAGES_DYNAMIC = "gdk-pixbuf-loaders-*" and recipes which
RDEPEND on some gdk-pixbuf-loaders-xxx package. To select between these
you need to set a PREFERRED_PROVIDER.
These are specified in the PN namespace so the locgical conclusion is
that setting PREFERRED_PROVIDER_gdk-pixbuf = "gtk+" should work. It
doesn't and instead checks crazy things.
The code was correctly finding the two possible providers, gtk+ and
gdk-pixbuf. It was however only accepting PREFERRED_PROVIDER_gtk+
= "gdk-pixbuf" to resolve this problem which reads as the exact
opposite to what was wanted.
This patch changes the code to do something that makes sense. I suspect
that before these changes it was pretty much a null operation rubber
stamping the single provider case. For Poky at least it exposes a few
cases where -nativesdk recipes were providing the same things as their
normal counterparts but these are genuine bugs in the metadata.
I've also attempted to make the multiple provider error message human
readable as I counldn't understand it and I doubt anyone else could
either.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For certain tasks, we need additional information in build stamp file
other than the task name and file name. stamp-extra-info is introduced as
a task flag which is appended to the stamp file name.
[Code simplifcations/tweaks from Richard]
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Even when a variable was whitelisted, any dependencies of that variable
could still creep into the task hash due to the way the whitelisting
code worked. This patch changes thing to ensure that when whitelisted,
that whitelisting applies to the variable and any dependencies it has.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We had a logic inversion that meant we where dropping quite a
significant number of events on the floor.... Fixed!
Signed-off-by: Joshua Lock <josh@linux.intel.com>
It's unlikely that someone wants to close the progress dialog
yet leave the UI (and BitBake process) running, so hook up
the progress dialogs delete-event to exit gtk.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
The recent change to Queue up events before the UI is spawned (in
26eda93337) broke the xmlrpc server because the
uievent implementation of BBUIEventQueue expects pickled strings for its
queue_event() method.
This is because the RPC exposed event.send() method must accept pickled
strings, but for xmlrpc event.send() is just mapped to queue_event().
Work around this by adding a send_event method which unpickles strings and
hands them off to queue_event() which can then be used for the remapping.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
It turns out that while log filters added with addFilter are only associated
with that logger, and not its children, handlers are inherited, and handlers
can be filters. So, let's add filtering to our existing LogHandler class
which dispatches our log records as bitbake events.
(Bitbake rev: 0153ace246e7c88366f45c8f035a2b4505a1c115)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
we should cache SRCREV whenever possible, the only exception is
when SREREV is auto rev. so change the logic to only set __BB_DONT_CACHE
at SRCREV = "${AUTOREV}" case
Signed-off-by: Yu Ke <ke.yu@intel.com>
Current fetcher has annoying "SRCREVINACTION" deadlock,
which occurs when SRCREV=${AUTOREV}=@bb.fetch.get_srcrev():
get_srcrev()->setup_localpath()->srcrev_internal_helper()
->evaluate SRCREV->get_srcrev()
current fetcher resolve the deadlock by introducing a
"SRCREVINACTION" condition check. Althoguh it works, it is
indeed not clean.
This patch use antoehr idea to break the deadlock: break
the dependency among SRCREV and get_srcrev(), i.e. assign
a specific keyword "AUTOINC" to AUTOREV. when Fetcher meet
this keyword, it will check and set the latest revision to
urldata.revision. get_srcrev later can use the urldata.revision
for value evaluation(SRCPV etc). In this case, SRCREV no longer
depends on get_srcrev, and there is not deadlock anymore.
Signed-off-by: Yu Ke <ke.yu@intel.com>
FetchData has some fetch method specific data, and only fetch method knows how
to initialize it. originally it is mostly initialized in Fetch.localpath().
But now there is requirement to call Fetch.latest_revision() before
Fetch.localpath(), thus require another earlier place for initialization. so
urldata_init is introduced for this purpose. it will be called in FetchData:__init__
and make all the Fetch functions useable after that.
Signed-off-by: Yu Ke <ke.yu@intel.com>
bb.fetch2 is copied from bb.fetch, and has many bb.fetch referrence.
Fix these referrence with bb.fetch2 referrence
Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We will be needing this information to improve the tracebacks of python code
from the metadata, as well as to give the user information about where
variables were defined, so they know how it ended up the way it is.
(Bitbake rev: 9615c538b894f71a2d1a0ba6b3f260db91e75786)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Errors can result from these expansions, but for skipped recipes, we
shouldn't care about those failures. This fixes the same issue which
Richard Purdie fixed in poky, commit 847b717.
(Bitbake rev: 96ee6840010c1ae1080e6bf7ff0f4eb2d361e84b)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This cleans up the knotty console messages to be a lot quieter and cleaning,
in keeping with the expectations of most users.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
[BUGID# 645], modify the emit_var()
1. Added "#" to the beginning of each line if the comment contains
multiple lines.
2. Added "\" to the end of each line if the shell variable value
contains multiple lines.
Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
The current parameters are not useful to the stampfile generator function
as they can't uniquely define a task. This updated things so the
parameters can identify unique tasks.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Move stamp file deletion out of the internal stamp helper function
* Add a new function to return the path to a stamp for a given task
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We can use the string split method for this instead.
(Bitbake rev: aa9646717b3ee1006628246a7c495f601e62391c)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This ensures that when a failure occurs very early on in bitbake startup, the
message formatting ematches that used by the UIs.
(Bitbake rev: c8ff0fd3e9f050a668f1a069cf37ee37db3664fa)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I think this is less confusing, and avoids needing to know about the *range*
of logging levels, instead simply asking what we really want to know.
(Bitbake rev: dc2264387617586b5c0a61e126c75edde5e7abcd)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The various alternative UIs have been updated to once again be functional
with the latest bitbake internals. Each of the UIs still have much room for
functional improvement.
In particular, they have been updated to:
- interact with the new process based server
- handle the current set of events and notifications fired from the server
and its associated subsystems
(Bitbake rev: b947e7aa405966262c0614cae02e7978ec637095)
Signed-off-by: Bob Foerster <robert@erafx.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
python tasks calling shell functions using exec_func() would show the log
file as /dev/null. It makes most sense for all the task logging to be setup
centrally by exec_task(), at least with the current code base in Poky.
This commit will need discussion in relation to upstream bitbake and the
IO redirection could be better handled using a context manager (although
task contexts shouldn't ever nest).
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Think this got inadvertantly dropped when switching to the new API.
(Bitbake rev: 628c5159d1151b89f2b7210c8819489e8dc9a84d)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Rather than having to run .addDomain() and then .getValue(domain, key),
.setValue(domain, key), etc, now it just works as mappings.
As an example:
setValue(domain, key) -> persist[domain][key] = value
It also arranges things so we can construct objects of this type using any
arbitrary filename/path for the sqlite3 database, rather than being so
tightly bound to the metadata.
(Bitbake rev: d9e8b8af308ae871efdc8ef0782be30af8c1f894)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Caching the database connection can cause serious issues if it results in
multiple processes (e.g. multiple tasks) simultaneously using the same
connection.
This reverts commit 8a6876752b90efd81d92f0947bfc9527d8260969.
(Bitbake rev: 60b9b18eafad5ac46c7cf1048d749d673c2ee0ad)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>