Commit Graph

10457 Commits

Author SHA1 Message Date
Chris Larson c8d2dad049 Rename the ui 'init' method to 'main'
As these may run the UI in a blocking fashion and then return the exit code,
'init' was an inappropriate name, and 'main' is more appropriate.

(Bitbake rev: 4d081a0ed759bd526ab01849d650bd9e8d80ddd1)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:47 +00:00
Chris Larson 72c6953488 cooker: use re match, not search in re_match_strings
We want to match the requested pattern at the beginning of the string,
otherwise things behave in an unintuitive manner wrt ASSUME_PROVIDED (e.g.
ASSUME_PROVIDED += "gtk+" will also assume foo-gtk+ is provided), and the user
can always use '.*gtk+' to get the old behavior.

(Bitbake rev: 5670134ab2eb573d39df3c3231677cdb1a1dfc72)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:47 +00:00
Chris Larson 431160caf1 taskdata: use 'any' in re_match_strings
(Bitbake rev: e48e9a2150ee76aaf151f6d5bc9e86e6ae4de514)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:47 +00:00
Chris Larson 246bd18613 cooker: add shutdown/stop methods
(Bitbake rev: fc64eff03fe1f41e59128cb3df0ef2161e24a8cb)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:47 +00:00
Chris Larson 825e5045f4 cooker: merge cookerState and cookerAction
(Bitbake rev: c7c8945ef7ca9465312e630b7fa5f0a87ac8b6c7)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:47 +00:00
Chris Larson ac4d926f41 cooker: use a pool, abort on first parse error
(Bitbake rev: 9caf65e79f95fe0045e727391e974c4c1e7411ff)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:46 +00:00
Chris Larson e5624a4ed3 knotty: use enumerate for task waiting
(Bitbake rev: 89ce8df075ac8c9a5478c86405e6e6b60346a51c)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:46 +00:00
Chris Larson 7ffd6f88b8 knotty: shift non-interactive progress into a class
(Bitbake rev: c3d005cbbae3d56da9926666cfb1501c2bf96ea7)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:46 +00:00
Chris Larson 7ea3c96938 Fix the <100 recipe progress fix
(Bitbake rev: 424428a764651183218f9cc93bc05496867aa5de)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:46 +00:00
Chris Larson 1e3b83f96b Error more pleasantly when trying to use python2.4
(Bitbake rev: 57402ee727c8c1d8f5017534fb0f5e0b20aaaa5b)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:46 +00:00
Chris Larson 7481c11e34 Don't show a traceback for a python version error
(Bitbake rev: 44c325306e55336b7b416fd88b39874012a3f603)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:46 +00:00
Chris Larson c4e07bc033 cooker: don't choke on <100 recipes to parse
(Bitbake rev: 2be5169a01e57bcfcb1f75cbd6bac3bf0f230edd)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:45 +00:00
Chris Larson b8aedaa6b5 cooker: no cached in progressbar and add ETA
Rather than updating the progress bar based on the recipe being processed
(whether cached or parsed), consider only parsed recipes.  This reduces the
instability in progress rate introduced by the cached entries, and allows the
ETA to be resurrected and be a bit more useful.

(Bitbake rev: 618480f7739f6ae846f67a57bee5a78efb37839d)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:45 +00:00
Chris Larson b890c19a33 cache: change to more incremental format
(Bitbake rev: 4fe4ffbef3885887c97eebe021edc3f23feab9ea)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:45 +00:00
Chris Larson 5bff22988c cooker: pass back child exceptions to the server
(Bitbake rev: 0f68f8bcd0e0aa944f76f88a4a85c9bcc1e42bee)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:45 +00:00
Chris Larson 8faec0b672 vercmp: don't choke on empty first version components
(Bitbake rev: e75aa94e9477933c5a40021b2a8e844db54f29da)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:45 +00:00
Chris Larson 6810622484 Use __file__, not sys.argv[0]
(Bitbake rev: 97e92abe49663eee189c89c1dc91fe69891faf73)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:45 +00:00
Chris Larson 8a938d567f cache: ensure 'pn' is included in the pkgvars
(Bitbake rev: cefc28a2aa1e5703119dacfd885c8c159a1e47a3)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:44 +00:00
Chris Larson a92caf45c8 knotty: drop the ETA from the progressbar for now
Currently, the progress bar is an indication of the processing of our recipes,
which includes loading the cache file, then for each recipe, either adding the
existing cached information to the CacheData or parsing the recipe from disk.
These tasks clearly take different amounts of time, so the ETA is unreliable
today.  We'll resurrect this functionality after we revamp the progress
handling, fully incorporating the load of the cache file.

(Bitbake rev: 80867372dcbef91ebaf7d77a77ca871741dd3f74)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:44 +00:00
Chris Larson 7a7e2f4e59 cooker: ensure that the cache sync completes
Without explicitly joining the thread, it's possible for the process to end
(e.g. after a bitbake -p) and kill off the thread without waiting for it to
exit cleanly.  So, register the thread join with atexit.

(Bitbake rev: 97ce57e6f860d3e6f34cc7a603ed1eeac4f423d3)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:44 +00:00
Chris Larson f4a06aac98 cooker: get number of threads in constructor
(Bitbake rev: e7fd259d1528f48b0812fb7b0a54dd6d5d22f4a9)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:44 +00:00
Chris Larson a519031f23 cooker: don't add info for skipped recipes
(Bitbake rev: ac596d55de4ad86e1cdc84dabfea1350dd54efd0)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:44 +00:00
Chris Larson 60226ff35c cooker: show progress bar before initializing the cache
This ensures that the time spent loading the cache from disk occurs with the
progress bar up.  Though the progress bar stays at 0% during this period, I
think this is an improvement over the multi-second stall which occurred
previously before the progress bar came up.  Ideally, we'd integrate cache
loading from disk into the progress display, but this is a first step.

(Bitbake rev: f6d0a5c219f9deb84f702450d30d868ba6271f77)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:44 +00:00
Chris Larson 6cd15a1ea0 cooker: save progress chunk value (total/100)
(Bitbake rev: 09333737cbeeb9875d938521ddcd519fc808bcc3)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:43 +00:00
Chris Larson 739bb5a2d1 cooker: stop loading the cache for -b
Previously, the cache was actually being loaded from disk twice whenever using
-b or -e -b.  This also moves the bb_cache instance into the CookerParser, as
it's not needed by the cooker itself at all.

(Bitbake rev: dd0ec2f7b18e2a9ab06c499b775670516bd06ac8)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:43 +00:00
Bob Foerster 065da895d2 cooker: don't fire unnecessary parse progress events
(Bitbake rev: 7cf7d5f14405681496fced3640a50a20ef1acac1)

Signed-off-by: Bob Foerster <robert@erafx.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:43 +00:00
Chris Larson 7846d8db11 cache: don't add info to cache if cache is disabled
(Bitbake rev: f12bb303f3d86a68d0b3dda1112dd654b9251704)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:43 +00:00
Chris Larson 7c9444e9a5 cache: sync the cache file to disk in the background
This version uses a thread rather than a process, to avoid problems with
waitpid handling.  This gives slightly less overall build time reduction than
the separate process for it did (this reduces a -c compile coreutils-native by
about 3 seconds, while the process reduced it by 7 seconds), however this time
is quite insignificant relative to a typical build.

The biggest issue with non-backgrounded syncing is the perceived delay before
work begins, and this resolves that without breaking anything, or so it seems.

(Bitbake rev: 5ab6c5c7b007b8c77c751582141afc07c183d672)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:43 +00:00
Chris Larson 95d2f56126 Simplify cache syncing
Rather than adding nocache items to the cache, then copying the cache and
removing them to sync it, don't add them in the first place.  Also use 'with'
for the cachefile.

(Bitbake rev: 343b6f6255ad020c39e30742175a241f0859a5a6)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:43 +00:00
Chris Larson 9ffbd9fe27 Experimental usage of the 'progressbar' module
(Bitbake rev: 64feb03bc2accecb49033df65e0a939ef5ab5986)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:42 +00:00
Chris Larson 32ea766871 Implement parallel parsing support
This utilizes python's multiprocessing module.  The default number of threads
to be used is the same as the number of available processor cores, however,
you can manually set this with the BB_NUMBER_PARSE_THREADS variable.

(Bitbake rev: c7b3ec819549e51e438d293969e205883fee725f)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:42 +00:00
Chris Larson 570bec37a8 cache: don't write out the cache unnecessarily
If the only recipes's we reparsed this run were those flagged as not to be
cached, there's no point in re-saving the cache, as those items won't be
included anyway.

(Bitbake rev: 1e0c4dbcbec886a30b89f8b4bb365c3c927ef609)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:42 +00:00
Chris Larson acca344057 cache: create and use a RecipeInfo class
This class holds the particular pieces of information about a recipe which are
needed for runqueue to do its job.

By using it, I think we improve code clarity, reduce method sizes, reduce
overuse of primitive types, and prepare for parallel parsing.  In addition,
this ditches the leaky abstraction whereby bb.cache attempted to hide the
difference between cached data and a full recipe parse.  This was a remnant
from the way things used to be done, and the code using it had to know the
difference anyway.  If we choose to reimplement caching of the full recipes,
we can do it in bb.parse, in a completely transparent way.

(Bitbake rev: 992cc252452221f5f23575e50eb67528b2838fdb)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:42 +00:00
Bob Foerster c6328564de Prefer xrange over range for small performance gain.
range() allocates an actual list when called.  xrange() is just an iterator
and creates the next range item on demand.  This provides a slight
performance increase.

In python 3, range will do what xrange does currently, but the upgrade will
be handled by the 2to3 tool.

(Bitbake rev: 73b40f06444cb877a5960b2aa66abf7dacbd88f0)

Signed-off-by: Bob Foerster <robert@erafx.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:42 +00:00
Chris Larson e81fc749f3 Revert "svn fetcher: warn people to switch to SRCREV"
1) too spammy
2) can be implemented in the metadata instead

This reverts commit 8da9744fcdf856abebcfbe9e3bc1b8cf07bc317b.

Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:42 +00:00
Chris Larson 66b364cf8e cache: make loadDataFull a classmethod
(Bitbake rev: 16a9d689e61c35cfca94bbecd9772eab3c5072ba)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:41 +00:00
Bernhard Reutner-Fischer 81a81a163c svn fetcher: warn people to switch to SRCREV
as noted by rp in ac00ca89a4e43cd4f38ba86455079d31be78e644

(Bitbake rev: 8da9744fcdf856abebcfbe9e3bc1b8cf07bc317b)

Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:41 +00:00
Bernhard Reutner-Fischer 4df0d6adca fetch: be more pythonic
no functional changes

(Bitbake rev: e88834fb7c6821cc29c12d296f2edd51f6eb3746)

Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:41 +00:00
Bernhard Reutner-Fischer b7d667f252 fetch: use os.path.join
(Bitbake rev: c360b01df18d90a513a3d61d395f905102e7568e)

Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:41 +00:00
Bernhard Reutner-Fischer 87b6f7d27a fetch: add common helper _strip_leading_slashes()
Several fetcher need a way to strip leading slashes off a local path.
This helper-function consolidates all such occurances.

(Bitbake rev: 823a02185ed109054c6c1ae366221aaed0353f24)

Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:41 +00:00
Bernhard Reutner-Fischer d0d67a9fe2 fetch: rename suppports_srcrev to supports_srcrev
osc had it already spelled correctly?!

(Bitbake rev: b8bb4433de7a981c6826173e926ca34705c4ac70)

Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:41 +00:00
Chris Larson 814c80f53c cache: Add debug msg for a nonexistant dep file
If a recipe depends on a file, and that file is out of date, we show a
message, but if that file was removed, we do not, until now.

(Bitbake rev: 67984ba0ac2db79874541bc031f2e3e9ff7a6c32)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:41 +00:00
Chris Larson 84f02831a7 cache: use new style classes
(Bitbake rev: e084430446be2544dd1a6b627088f888c37cc7f0)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:40 +00:00
Chris Larson c4d939079e cache: pyflakes/pep8/pylint cleanup
(Bitbake rev: 06420ff839ffc37de3e42474b8b0e47c4608a985)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:40 +00:00
Chris Larson 77d52cb1b8 cache: use set for clean, checked
(Bitbake rev: 3a776ef947e23b1a04bdd9e78a3a0c4616249b0e)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:40 +00:00
Chris Larson f207bb7c93 cache: make use of defaultdict
(Bitbake rev: 7171019b11ad656d0edb979564941fcf92ab0a02)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:40 +00:00
Chris Larson 9c7c18d06f Pre-explode rundeps/runrecs in CacheData
(Bitbake rev: 0c8e0f3191252ccedb83c431c48e5c7d905e2717)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:40 +00:00
Chris Larson 3a2176fe73 persist_data: handle locked db for SELECT
Parallel processes interacting with the persist_data db can quite easily
explode without this.

(Bitbake rev: b3d5432cff0ff28f4c8a5bcf10efa3e383b4fd4d)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:39 +00:00
Bernhard Reutner-Fischer f565258368 BBHandler: use os.path in inherit()
(Bitbake rev: 5b85de2c71973ba490b95a5d9ab634635f395142)

Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:39 +00:00
Bernhard Reutner-Fischer b44100eafc BBHandler: simplify supports()
(Bitbake rev: 858d704d713d15bf97053eb1374758c74b0d8874)

Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04 14:46:39 +00:00