self.appendlist is a dict and as such unordered. This can lead to cases
where appends with different names (e.g. x_%.bbappend vs. x_123.bbappend)
can be reordered in application which in turn reorders the variables
that those bbappend files might touch. Reorderd variables changes the sstate
cache signatures causing real world issues.
To avoid this, use a list for the append files instead.
This patch is conservative and just adds a new data structure alongside
the existing one and uses it to resolve the core issue. Later patches
(post release) can handle some of the wider but less problematic ones
(e.g. issues in bitbake-layers flatten).
[YOCTO #7511]
(Bitbake rev: f980f060cd0d1e7fe5011f3c325c1b254f05eccf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The recent inotify changes are causing a 100% cpu usage issue in the
idle handlers. To avoid this, we update the idle functions to optionally
report a float value which is the delay before the function needs to be
called again. 1 second is fine for the inotify handler, in reality its
more like 0.1s due to the default idle function sleep.
This reverts performance regressions of 1.5 minutes on a kernel build
and ~5-6 minutes on a image from scratch.
(Bitbake rev: 0e0ba408c2dce14a0fabd3fdf61d8465a031495b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch fixes a bug where if the build is force stopped,
subsequent clients cannot connect to the server due to
unnecessary limits on setFeature.
Additionally, we make sure that the state is properly reset
even if the BuildCompleted event firing excepts for some reason.
(Bitbake rev: 0b66b05169688aa4ddc4c54d175bb961b2f27fec)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The inotify facility monitoring changes to the config files
could be overwhelmed by massive changes to the watched files
while server is running.
This patch adds verification the notification watches to the
server idle functions, in addition to the cooker updateCache
command which executes only infrequently, thus preventing
overflowing the notification buffer.
[YOCTO #7316]
(Bitbake rev: 996e663fd5c254292f44eca46f5fdc95af897f98)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've had versioned dependency support in LAYERDEPENDS for quite a long
time, but I can say with pretty good certainty that almost nobody has
used it up to now because it was too strict - the specified version had
to exactly match the version in your configuration or you would get an
error; there was no "greater than or equal" option, which is usually
what you will want given that LAYERVERSION does get bumped from time to
time.
However, users mismatching layer branches and then having their builds
fail later on with some incomprehensible error is still a pretty common
problem. We can't simply use the git branch because not everyone is
always on a branch and the branch names don't always match up (and
that's not an issue). To provide a practical means to address branch
mismatching, I have reworked LAYERDEPENDS version specifications to use
the more familiar "dependency (>= version)" syntax as used with package
dependencies, support non-integer versions, and clarified the error
message a little. If we then take care to bump the version on every
breaking change, it is at least possible to have layers depend on these
changes when they update to match; we can now even support a major.minor
scheme to allow retrospectively adding a version limiter to old branches
when a new branch is created and yet still allow the old branch minor
version to be bumped if needed.
Fixes [YOCTO #5991].
(Bitbake rev: 408be9cdf2b1e32e64ea488d8051a546fb54c144)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Users expect operators like _remove to work on this variable. We need
to use expanded_data to ensure this happens correctly.
[YOCTO #7135]
(Bitbake rev: cc4c8478fc547ea0ebf827a8d319496b39f25684)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
event_data would be better named expanded_data, then we can widen its scope
to other places in cooker where we need to access an expanded data store.
We certainly don't want multiple expanded data stores.
(Bitbake rev: 1a3c1c9203e1a1452314954f1cfd771e5c1ce89b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Trap the ENOSPC error and translate it into a human readable error
message, which is good for debugging.
(Bitbake rev: 2b084dff6ff0d274fbbf7ab07022507f7249e427)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We currently add crazy numbers of watches on files. The per user limit is 8192
by default and on a system handling multiple builds, this can be an issue.
We don't need to watch all files individually, we can watch the directory containing
the file instead. This gives better resource utilisation and better performance
further reverting some of the performance regression seen with the introduction
of pyinotify.
(Bitbake rev: a2d441237916a99405b800c1a3dc39f860100a8c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the environment changes, we need memory resident bitbake to adapt to those
changes. This adds in functionality to handle this alongside the configuration
option handling code. This means that the common usage:
MACHINE=X bitbake Y
now works with the memory resident server.
(Bitbake rev: 4d1343010da757a0c126bc22475354da44aaf8e3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Benchmarks show that the introduction of pyinotify regressed
performance. This patch ensures we only call the add_watch() function
for new entries, not ones we've already processed which does improve
performance as measured by "time bitbake -p".
This doesn't completely remove the overhead but it does substantially
reduce it.
(Bitbake rev: 493361f35f6cc332d4ea359a2695622c2c91a9c2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We try and add watches for files that don't exist but if they did, would influence
the parser. The parent directory of these files may not exist, in which case we need
to watch any parent that does exist for changes. This change implements that fallback
handling.
(Bitbake rev: 979ddbe4b7340d7cf2f432f6b1eba1c58d55ff42)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Memory resident bitbake has one current flaw, changes in the base configuration
are not noticed by bitbake. The parsing cache is also refreshed on each invocation
of bitbake (although the mtime cache is not cleared so its pointless).
This change adds in pyinotify support and adds two different watchers, one
for the base configuration and one for the parsed recipes.
Changes in the latter will trigger a reparse (and an update of the mtime cache).
The former will trigger a complete reload of the configuration.
Note that this code will also correctly handle creation of new configuration files
since the __depends and __base_depends variables already track these for cache
correctness purposes.
We could be a little more clever about parsing cache invalidation, right now we just
invalidate the whole thing and recheck. For now, its better than what we have and doesn't
seem to perform that badly though.
For education and QA purposes I can document a workflow that illustrates this:
$ source oe-init-build-env-memres
$ time bitbake bash
[base configuration is loaded, recipes are parsed, bash builds]
$ time bitbake bash
[command returns quickly since all caches are valid]
$ touch ../meta/classes/gettext.bbclass
$ time bitbake bash
[reparse is triggered, time is longer than above]
$ echo 'FOO = "1"' >> conf/local.conf
$ time bitbake bash
[reparse is triggered, but with a base configuration reload too]
As far as changes go, I like this one a lot, it makes memory resident bitbake
truly usable and may be the tweak we need to make it the default.
The new pyinotify dependency is covered in the previous commit.
(Bitbake rev: 0557d03c170fba8d7efe82be1b9641d0eb229213)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the cooker is in an error state, we shouldn't continue to try parsing.
This fixes an issue where an invalid PR server is detected when bitbake
is started and ensures bitbake exits cleanly rather than hanging.
[YOCTO #6934]
(Bitbake rev: 294bb9cad294423d4f8998405ceff58655f12660)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a BBFILE_PATTERN_IGNORE_EMPTY variable to allow ignoring the fact
that a regex specified in BBFILE_PATTERN for a particular collection
doesn't match any recipes. This will be used in OpenEmbedded in the
workspace layers created by "devtool" which may not always contain any
recipes (which is not cause for warning the user).
(Bitbake rev: 19c74aaa2836a88d666f0032452fac521689ab6b)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds a "-w/--write-log" option to bitbake
that writes an event log file for the current build.
The name of the file is passed as a parameter to the "-w"
argument. If the parameter is the empty string '', the file
name is generated in the form bitbake_eventlog_DATE.json,
where DATE is the current date and time, with second precision.
The "-w" option can also be supplied as the BBEVENTLOG
environment variable.
We add a script, toater-eventreplay, that reads an event
log file and loads the data into a Toaster database, creating
a build entry.
We modify the toasterui to fix minor issues with reading
events from an event log file.
Performance impact is undetectable under no-task executed builds.
(Bitbake rev: 1befb4a783bb7b7b387d4b5ee08830d9516f1ac2)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if an invalid PR service is selected the server will error
with a traceback. This is because its set into the error state and the
setFeature code will then fail since its not in the initial state.
Modifying the featureset in the error state is acceptable, we just need
to ensure we don't trigger a reset, that would happen from whichever
code handles the error.
[YOCTO #6934]
(Bitbake rev: c52841445d8db8f84c4da34203b195fea5874247)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
1) Run "bitbake recipe" in the terminal
2) Close the terminal while building
3) $ ps aux | grep bitbake-worker
There will be many processes, and they will keep the resources (e.g.,
memory), and won't exit unless kill or kill -9.
(Bitbake rev: 40d2ae0723de2bf5fee343faafb4afda40546839)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
loginfo "op" was being set to invalid values in saveConfigurationVar it was
working because set is a python global but append is not. This replaces both
the "op" with strings and removes the crash when calling appendConfigurationVar
(Bitbake rev: 2a8e847de85546d43600a561f9c63aa36bd69222)
Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using options like -k, -f, -v and so on with the memory resident
server, they'd currently only be set on the initial values passed to
the original command. This ensures they now match those specified
on the commandline for the options where this makes sense.
To make this work, a command to update the options on the server side
is required so this is added.
[YOCTO #5292]
(Bitbake rev: 1c75cc4d0c8b606c1fe76e6bf60bf6a32298b105)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
since python-2.5 string.endswith() takes a tuple
(Bitbake rev: 86a67a1fd4244da9343dbf14deed1ad0d3003f32)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The re module was imported at the top, no need to do it again.
(Bitbake rev: c29c2370635075949c75937c237602600f4f10dc)
Signed-off-by: Namhyung Kim <namhyung@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 1cd369883469747a8158826bb8d67dcca2a8577f)
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This solves a problem of custom images which inherit a base
image with IMAGE_BASENAME overwritten in their recipe by a
different value than its default one: ${PN}.
The value of IMAGE_BASE causes a crash when hob will try to
create symbolic links to the resulting images from the deploy
directory, because it will look for names similar to
<original_recipe_name>-edited-timestamp-machine.rootfs.*
which might be different from the actual resulting image.
The solution is to simply overwrite IMAGE_BASENAME in the
custom recipe to the default value in the case IMAGE_BASENAME
is found in the base recipe.
Some recipes which were affected by this problem are those
from meta-fsl-demos (e.g.: fsl-image-test).
[YOCTO #6017]
(Bitbake rev: e42ee93519000f827be49659b6b5fb7717b3d592)
Signed-off-by: Marius Avram <marius.avram@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch makes sure that the setFeatures command is marked
as read-only and that it can only run if the cooker is in
the initial state.
Additionally, remove logging from the XMLRPC module in favor
of sending the exception to the client for easy processing.
[YOCTO #6089]
(Bitbake rev: f0a1a3e24757f7658d272035620465f92a3e4c3c)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the featureset didn't match the defaults, we'd pay the price of two
base configuration parses which showed up adversely in the performance
benchmarks. This also passes the feature set into the cooker creation so
in the common case we don't have to reset the server. This speeds up
both knotty and hob startup. If the featureset doesn't match, the system
will reset as before, this just streamlines the common case.
(Bitbake rev: 1249543c4dbf3edeac033d888497864cfc807a4e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the change to more optimal default featureset behaviour, a race was
exposed by hob where the code may try and change self.data before it
exists. This change avoids that.
When the datastore is created, the cooker configuration is used so
data tracking is correctly handled regardless.
(Bitbake rev: 9d8f7efbc39d64124936ccaeb3c47a112e595d78)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sanity tests are currently a pain as its hard to control when they run. This results
in issues where for example the bitbake -e output is not useful as the sanity tests
prevent it from executing. The sanity tests should run later than the base configuration.
This patch changes the sanity tests to always be event triggered with the option of
returning either events on the status, or raising errors. A new cooker feature is used
to change the behaviour depending on the controlling UI.
This does need a change to sanity.bbclass in the OE metadata but its worth the pain
for the increased flexibility and control this offers UIs and the improvement to the
user experience.
(Bitbake rev: 32e171bcc92c6e27fefee971e8714ddf8e1a8ac1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When a variable was removed from a configuration file it was not
removed from memory. This also had the effect of not allowing
to set a new value for the same variable with saveConfigurationVar.
(Bitbake rev: 30cd1fab6633aaf50ef53eefccc6d69d598eb293)
Signed-off-by: Marius Avram <marius.avram@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixes an issue in hob which happened when the local.conf file was
modified externally by appending "eclipse-debug" to the IMAGE_FEATURES
variable. The reason of the problem is that some IMAGE_FEATURES are
not available in the image.bbclass file and they are declared in the
core-image.bbclass. Now a default hob image will inherit core-image.
[YOCTO #5711]
(Bitbake rev: 81413d94f40f58d790d7a7dc4259108f9c5d4fc0)
Signed-off-by: Marius Avram <marius.avram@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the cooker receives a SIGTERM it currently hangs using 100% CPU,
This patch adds in an intercept for the event and puts the cooker into
shutdown mode allowing it to exit cleanly/safely and avoiding the hang.
(Bitbake rev: 00c22434123739b0819b31d7b1d353901a3e12da)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a command to allow backends to set particular 'features' on the
cooker (server).
(Bitbake rev: f547d6ec6cfd677d71fa96dd3c69823c00dc6c69)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It must be checked first if parser has the attribute 'shutdown' when
user hits Stop button and the forceshutdown state is given.
[HOB #5579]
(Bitbake rev: 46943b442ea4fa778f70590b6dcce483595efaf8)
Signed-off-by: Irina Patru <irina.patru@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently its near enough impossible to tell bitbake to run target X,
task Y and target A, task B. We could hack various parts of the API
around but it would mean incompatibilities.
An alternative is to accept the syntax "<target>:do_<task>" as a target
name. The default task would be used where the task is unspecified.
This has the advantage that its neat/clean code and works from all
current APIs including the commandline.
(Bitbake rev: 55f6bee3114e582333a1784caeddb197b9163d02)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 34afbdd0fc809b8fb20696aeef3e6a61d6812e16)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
initConfigurationData and loadConfigurationData are similar functions, the only
reason for them appears to be to be able to reset the pre/post configuration
files. The current code is confusing and unmaintainable.
Instead this patch creates a new Sync command which allows these to be explicitly
set. The init and load functions can then be merged into one. There is then no
need for a parseConfiguration command, we can simply reset the server to have the
settings take effect.
The reset fuction is not an instant value return and triggers an event so it should
be an Async command, not a sync one.
The number of calls for the set pre/post command is probably higher than it
need be but someone with more familiarity with the hob code base can probably
figure out the right places its needed (maybe just init_cooker?).
(Bitbake rev: bae5210d7e048022f083361964ebec7daf1608f7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The idea is to build on the --status-only option for bitbake and
expose a mechanism where the oe init scripts can easily switch between
memres server and the non-memres server.
In the case of the standard oe init script the following
can shut down the server:
if [ -z "$BBSERVER" ] && [ -f bitbake.lock ] ; then
grep ":" bitbake.lock > /dev/null && BBSERVER=`cat bitbake.lock` bitbake --status-only
if [ $? = 0 ] ; then
echo "Shutting down bitbake memory resident server with bitbake -m"
BBSERVER=`cat bitbake.lock` bitbake -m
fi
fi
A similar function can be used to automatically detect if the server
is already running for the oe memres init script. This new
functionality allows for the memres init script to be started in a new
shell and connect up to an alaready running server without seeing the
error of trying to start the server multiple times.
(Bitbake rev: b1803958de8d7c3c3279841e38604a08dc2316cc)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The 'bbappend in f' incorrectly compares the current recipe with the avaliable
bbappends recipes. This comparsion causes unrequested bbappend files to be
appended, e.g. in the case of 'libgcc_4.8.bb', the bbappends for 'libgcc_4.8.bb'
and 'gcc_4.8.bb' are added to the filelist (because 'gcc_4.8.bb' is contained in
the 'libgcc_4.8.bb' string) which in turn causes the gcc_4.8.bbappend files to
be appended to the libgcc_4.8 recipe.
This should be a 'bbappend == f' to match the previous implementation of this
function, such that if no wildcard is present the recipe names must match
exactly.
This issue was introduced by commit 31bc9af9cd56e7b318924869970e850993fafc5f,
which it related to [YOCTO #5411].
(Bitbake rev: 991cbeedbde8bd25ce08c669b1bfac8b99e33149)
Signed-off-by: Nathan Rossi <nathan.rossi@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There has been a continuing call for supporting wildcard in bbappend
filenames. The wildcard is actually allow matching of the name and
version up to the point of encountering the %. This approach will
allow for matching of the major or major.minor.
Exampes:
busybox_1.21.1.bb
busybox_1.21.%.bbappend will match
busybox_1.2%.bbappend will also match
if we update to busybox_1.3.0.bb the above won't match, but a busybox_1.%.bb
will.
[YOCTO #5411]
(Bitbake rev: 31bc9af9cd56e7b318924869970e850993fafc5f)
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch changes bitbake to log an error to the
command line when the build is interrupted via Ctrl-C.
This is needed to inform the user that not all tasks
required for the build have been executed, and
the build is not complete.
Internally, the Bitbake server will return a CommandFailed
event that will be logged by Toaster as build failure.
(Bitbake rev: 9a658e8b1511f1b9f91663f546f748fdfbc8965f)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to track the file where a configuration
variable was defined, this patch bring these changes:
* a new feature is defined in CookerFeatures, named
BASEDATASTORE_TRACKING. When a UI requests BASEDATASTORE_TRACKING,
the base variable definition are tracked when configuration
is parsed.
* getAllKeysWithFlags now includes variable history in the
data dump
* toaster_ui.py will record the operation, file path
and line number where the variable was changes
* toaster Simple UI will display the file path
and line number for Configuration page
There is a change in the models to accomodate the recording
of variable change history.
[YOCTO #5227]
(Bitbake rev: 78e58fed82f2a71f052485de0052d7b9cca53ffd)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the user explicitly asks to build a target that is listed in the
value of ASSUME_PROVIDED, show a warning mentioning that it will be
ignored.
(Bitbake rev: 65be09f5e2a4a41e65c9232e208d8154b822fc4e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Also removed some redundant file manipulation code
Based on patch sent by Stefan Stanacar <stefanx.stanacar@intel.com>
(Bitbake rev: e054c1e7c8581f66082fcdfb89769401ca6e78a3)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When building a single file, the cooker will recreate
the recipecache from scratch.
I suspect this is a remnant of past code, since:
* the current recipecache works fine
* the new recipecache will not have all the fields as
requested by HOB_EXTRA_CACHES setting
This patch disables recreating the recipecache, leading
to shorter times when building single build files
(-b option) and better compatibility with Toaster.
(Bitbake rev: 618d69b00075981b8553513130d7deb1aed61578)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Toaster needes to record extra data that needs to
be moved at the time of the dependency tree dump.
This data includes:
* layer priorities for recording in the layer section
* the inherit list for each PN which allows to determine
the type of the PN (regular package, image, etc).
This patch adds this data to the dependency tree dump.
(Bitbake rev: 7636aba37320aaf9b044d3832ddc21af51ccd69c)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if errors occur when starting the PR service, there is a race that
occurs since the UI runs various commands including starting builds before
processing the CookerExit(). By adding the error state and refusing to run
async commands in this mode, builds are prevented from starting and the
UI reaches the exit code with the system shutting down cleanly.
(Bitbake rev: 42fa34142ea685f91115a551e74416ca28ef1c91)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The patch that implements removeConfigurationVar method was made before
merging the patch that replaces "added by bitbake" with "added by hob".
This patch corrects this issue.
[YOCTO #5284]
(Bitbake rev: 42601a5edef0316767b952b162123534aa8fee18)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In Hob settings, there is a tab to add/remove extra settings. This
patch implements a way to "remove" variables from conf files, through
bitbake. But, to keep the history assigment of the variables synchronized,
instead of removing, it replaces the lines with blank lines.
[YOCTO #5284]
(Bitbake rev: bd720fb63cef6b399619b8fbcaeb8d7710f2d6df)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
MACHINE var is saved using early assignment operator.
Calling MACHINE=x bitbake core-image-... works properly.
Comment "#added by bitbake" is replaced with "#added by hob".
[YOCTO #5070]
(Bitbake rev: 2d0ec8ff083b636a6cf98de3278900eb95c3def6)
Signed-off-by: Valentin Popa <valentin.popa@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding the 'getAllKeysWithFlags' read-only command that will
return a dump of the global data state, together with specified
flags for each key. The flag list is passed in as the first
parameter to the command.
This will be used by UI clients to get the build configuration.
(Bitbake rev: 3e094da513e1220319288806cb76ddf804772afd)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The loaded cache modules may add extra attributes to
the recipecache, that will be populated by the cache
classes required by the UI. These attributes
will be used by the UI to display relevant information.
Adds cachefields cache class field to specify
for each cache class which attributes will be set
in the recipecache.
Adds code to automatically expand depends tree with the
fields exported by the extra cache class.
Fixes a cache field name in the HOB UI.
(Bitbake rev: 47c171005fb3803d936e65fcd4436c643883ae16)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding a CookerFeature that allows UIs to enable
receving a dependency tree once the task data has been
computed and the runQueue is ready to start.
This will allow the clients to display dependency
data in an efficient manner, and not recompute the runqueue
specifically to get the dependency data.
(Bitbake rev: 75466a53b6eece5173a9bfe483414148e4c06517)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Implementing feature set selection that allows a client
to enable specific features in the server at connection time.
Only enabling of features is supported, as there is
no way to safely remove data loaded into the cooker.
Once enabled, a feature will remain enabled for the
life of the cooker.
Client-server connection now supports specifying the feature
set required by the client. This is implemented in the Process
server using a managed proxy list, so the server cooker
will now load dynamically needed features based on what client
connects to it.
In the XMLRPC server the feature set is requested by
using a parameter for registerUIHandler function.
This allows observer-only clients to also specify features
for the server.
The server code configuration now is completly separated
from the client code. All hardcoding of client knowledge is
removed from the server.
The extra_caches is removed as the client can now specify
the caches it needs using the feature. The UI modules
now need to specify the desired featureSet. HOB is modified
to conform to the featureSet specification.
The only feature available is CookerFeatures.HOB_EXTRA_CACHES
which forces loading the bb.cache_extra:HobRecipeInfo class.
(Bitbake rev: 98e594837aab89ea042cfa9f3740d20a661b14e2)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The syntax used for checkPackages implies a copy is returned but it
did not do so. Make it so.
This is fixes universe builds where error messaages were being shown
but there should have only been warnings.
[YOCTO #5222]
(Bitbake rev: 97db2a0792d605f27d434bc6a4acce52857deee1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Clean-up to avoid duplication and promote code reuse to factor
taskdata creation into a common function.
[RP: minor tweaks]
(Bitbake rev: 468c221449290c4f196e87f7d8e23fcd7db86135)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Exiting from the server is antisocial, instead we should raise an exception. This
will correctly fail the current command and reset the server state. We use
the handled exception since for these conditions to occur, something was
already displayed to the user.
(Bitbake rev: dacc94bcace85a2e95aee2dccd8e680c59e4545f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After running a command on the server, it needs to reset to the initial
state. This ensures that subsequent clients start from a known state
and notice any configuration changes.
Ultimately we may want to do more than this buts a good start and better
than nothing.
(Bitbake rev: dd15648fc2654b8d7c3e00ea7ab3dbf04f24f24b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The shutdown state causes the server to finish what its doing, stop was
them meant to completely stop it. It doesn't mean the server is stopped
though. Renaming the current stop event for forceshutdown gives more
meaning to what it actually does. The stopped namespace then becomes
available to indicate a completely stopped server.
(Bitbake rev: 12e9d33bfae5294e3870dfd1202f63383ad05e92)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the cooker event data isn't rebuilt upon reset and the cache
configuration cannot be changed after init. These are both bad things
and this patch refactors the init/reset code so that it is possible
to reconfigure the server.
(Bitbake rev: 1193b8d76fcb6cb87e9ec135a2514370d7dd90ac)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This call only ever happens in cooker context now so we can drop the
nasty worker check from here.
(Bitbake rev: bc0b30199a8e3624c5b9914430adbcc7c6bd4497)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Cookerdata.findconfigFile method has a new parameter. Changed some calls.
(Bitbake rev: dce0f9d4afe0986e2dd0146944fc4ac9dde275e4)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
updateCache handles the logic for shutting down the parsing so we need
to call it for all cases when we're not running.
This fixes hangs if Ctrl+C is pressed during parsing.
(Bitbake rev: 552b8935dd2f9f11e8d5c08a597a7e966b891480)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Hob retrieves the list of recipes and packages using the IMAGE_INSTALL
variable, so a custom image should be saved using this variable.
Changed how the image is saved in a bb file
[YOCTO #4193]
(Bitbake rev: edf3f52c05d86d49b71770cdafde583213e2034d)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Saving an image that requires another image will cause issues when
the second on is removed. So, we have agreed to "require" only the images
from layers.
The functionality is implemented in bitbake, in order to be more abstract,
and it is used by Hob when an image recipe is saved.
[YOCTO #4193]
(Bitbake rev: 28296ca78507ba2e414eb136c81afee65a8e25e5)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It was necessary to append ${TOPDIR}/recipes/images to BBFILES.
Implemented the mechanism to append a value to a variable: a command and
the method in cooker.
[YOCTO #4193]
(Bitbake rev: 4aedbee90bd92395c2460a68702e6ede00e256c9)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When an new image is saved, the dialog for this action has
a field for the description. Changed how an image is saved, by
appending the DESCRIPTION variable at the end of the .bb file.
[YOCTO #4193]
(Bitbake rev: 5629007f2b984005e3a8ac5d9b71422cbc2f1409)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Modified generateNewImage function from cooker, in order to be used to
save a template in Hob.
Created a command to ensure that some dirs are created. The templates
(recipes) will be saved in {TOPDIR}/recipes/images folder.
Called these methods from Hob.
[YOCTO #4193]
(Bitbake rev: 96ffa00945c7eb09a0132fa47159aef3ef20fb3e)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In this case, the comment is appended to the end of the file.
Some text editors, do not place a '\n' to the end of the file
after saving it.
[YOCTO #4636]
(Bitbake rev: 2beb9589b1bd9773f587b4dc08afdfe50f4ea913)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As it is implemented now, when parsing the configuration files is
needed, it also needs an initialization.
Also, implemented a function to create an extra configuration files
which can be used as postfile/prefiles when parsing.
(Bitbake rev: 5fe23c1b51da14f58f8c483f43f30b48766a0913)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we reset, the current tracking value may change unexpectedly. This
patch ensures that the default value is preserved over a datastore reset.
(Bitbake rev: 0aa27598d9fdc8c3d18839b3ef93d0ba9698f761)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The command for reparse does nothing, so I have managed to remove it,
because it can be misleading. When a reparse is needed, it can be used
the parse function.
(Bitbake rev: 188eaba121789112ffeb1188f0984d23dfe8df4f)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the change to bitbake-worker we need to ensure the workers know
how to contact the PR service, the magic 0 port and singleton is
no longer enough.
(Bitbake rev: c761751e259bb8e940552a28794b45887b5a72d9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The reasons for combining these objects is ancient history, it makes
sense to clean things up and separate them out now. This follows on
logically from the configuration cleansups and leads well into the
bitbake-worker changes.
(Bitbake rev: 89ffd62661ebcf2a97ce0c8dfd5e4d5bfbe27de7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The collection of the extra caching data should not
be performed by the cooker, but supplied to it.
This patch will also streamline the code for launching servers
without a UI attached.
Based on a patch by Bogdan Marinescu <bogdan.a.marinescu@intel.com>
(Bitbake rev: f0b54280a6bce522508e4741e5f507bc284113a8)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The init function call was clobbering data hob relies on injecting, fix
this so the data injected by hob isn't overwritten.
(Bitbake rev: 64ca1aa1fd5f0473196b6cd805637fa644a92173)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Originally it seemed like a good idea to keep the parameters around. Having
seen this in real life use, its incorrect, we should pull all the data we need
into the cooker's configuguration and then use this to build the datastore.
Being able to just build the datastore from the parameters seemed like a good
idea but having a dummy cooker configuration object is now looking like
the better option.
This also fixes failures in hob since the parseFiles command can call
into cooker directly now and reset the configuration prefiles and postfiles
at will, rather than the indirect calls before which were breaking the datastore
(e.g. BBPATH wasn't set).
The cleanup this allows in tinfoil illustrates how this change makes more sense.
(Bitbake rev: f50df5b891bf318f12fc61c74adfcc626cc6f836)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Similarly to the execution context changes, establish better lifetime
management API of the class event handlers.
(Bitbake rev: 54e35a6cceead9521f8b1dacd48e55064e85c8bd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current execution context management for bitbake is ugly and the
use of a global variable is nasty. Fixing that is hard, however we
can improve things to start to establish an API for accessing
and changing that context.
This patch also adds in an explicit reset of the context when we reparse
the configuration data which starts to improve the lifecycle of the data
in setups like hob.
(Bitbake rev: 6c3281a140125337fc75783973485e16785d05a1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its confusing when an event handler in for example a bbclass receives
events for a recipe which doesn't use the class. This is due to the
class event handlers having confusing scope. Worse, the issue is not
deterministic and the events a handler will see depends on the parse
order.
To avoid these issues, remove the class event handler global scope
and only have class handlers recieve events for recipes using the
class.
(Bitbake rev: 7fb95f3d133e440d463d2faa7151c731f8e1ae96)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Having a global method scope confuses users and with the introduction
of parallel parsing, its not even possible to correctly detect conflicting
functions. Rather than try and fix that, its simpler to retire the global
method scope and restrict functions to those locations they're defined
within. This is more what users actually expect too.
If we remove the global function scope, the need for methodpool is reduced
to the point we may as well retire it. There is some small loss of caching
of parsed functions but timing measurements so the impact to be neglibile
in the overall parsing time.
(Bitbake rev: 4d50690489ee8dc329a9b0c7bc4ceb29b71e95e9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to have a memory resident bitbake and to allow task execution, we need
to be able to rebuild the base configuration without a cooker. This moves the
code into its own class so it can be built independently.
The interface is less than ideal here but I didn't want to add parsing methods
a subclassed DataSmart, at least until we've experimented further with this code
and are certain that makes sense. At the very least, the methods are ugly and need
cleaning up. Spliting the code out seems to be the right thing to do though and
should unblock various activities on BitBake so I believe this code is a step in
the right direction.
Based on a patch from Alexandru Damian <alexandru.damian@intel.com>
(Bitbake rev: 22a0b3cf73d2689db0c118b37aa7492632f8b0a7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Building up a set of actions for the server is tricky since we depend upon the
commandline but fall back to values from the datastore. We should be able to build
a datastore without a commandline and vice versa. Ultimately the UI should send
the commands to the server.
This patch amounts to code rearranging, moving the heavy lifting to the UI, though
a helper in the configuration option. This will need further cleanup/tweaking but
this should be the only update needed to the UIs. The code now queries the server
for any missing data should it need to.
This code allows various knowledge of configuration variables to move to the UI side
only, partcularly pkgs_to_build but also all the command specifiers. It should also
be possible to move cmd eventually, I'm just unsure if any callers call the commands
expecting this to default to something sane right now.
(Bitbake rev: 2dbbb1d51dafd4451fef8fe16f095bcd4b8f1177)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the UI and server configuration is one big incestuous mess. To
start to untangle this we creater cookerdata, a new module which contains
various confiuration modules and the code for building the base datastore.
To start with we add a ConfigParameters() class which contains information
about both the commandline configuration and the original environment.
The CookerConfiguration class is created to contain the cooker.configuration
options. This means we can transfer new paramters to the server over something
like XMLRPC and then build a new configuration from these on the server.
Based on a patch from Alexandru Damian <alexandru.damian@intel.com>
(Bitbake rev: 35bd5997e8d8e74bc36019030cc10c560a8134f9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
"status" is a relic of very early versions of bitbake, rename it to match what it now
means.
(Bitbake rev: db307fc68773868de2db614a061f51c5c6a90468)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
updateCache() gets called by command.py when needed and needs to be iterated
over. The calls in cooker.py are therefore just plain wrong/confusing now.
(Bitbake rev: 8b7c65bba466bb6773c56849074978ce2c956129)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its clear looking at the code the path argument has never been used and doesn't
work. Tweak the function to be a little more readable.
(Bitbake rev: d7047ceaf29b870ade3257e263e010f65aa8bc6e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The Cooker class is too large and needs to be split up into different
functional units. Splitting out the collections code into its own class
seems like a good place to start to try and disentangle things.
(Bitbake rev: ca1fcbb6e214c155a05328779d3d326e10c5eac0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The <> operator is deprecated, replace with !=.
(Bitbake rev: 4a43e58dd97ec6ea304eaa727c030973a5bc91b7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Current if the PR Service fails to start, bitbake carries on regardless or
hangs with no error message. This adds an exception and then handles it correctly
so the UIs correctly handle the error and exit cleanly.
[YOCTO #4010]
(Bitbake rev: 949c01228a977c3b92bfc0802f6c71b40d8e05b3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
So that the information added to cache_extra could
be accesed by hob, new variables were added in
the cooker.py.
(Bitbake rev: f2d5f4ca9ac82599c74838844f7e54e481e023d3)
Signed-off-by: Andrei Dinu <andrei.adrianx.dinu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
-extended cooker.py so that the extra information in cache_extra.py could
be used
(Bitbake rev: e6c2fa2f6669a4adf890a2ee794b100617223b7b)
Signed-off-by: Andrei Dinu <andrei.adrianx.dinu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Reverting the pool changes, terminate does not work reliably on
bb.compat.Pool :(
[YOCTO #3978]
This reverts commit 8af519a49a3374bd9004864ef31ca8aa328e9f34.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Reverting the pool changes, terminate does not work reliably on
bb.compat.Pool :(
This reverts commit 9ca8e513dc142dd7aefadeb6db2ccdb00f38b39f.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Multiprocessing catches these and handles passing them between processes
itself, we don't need to do it ourselves anymore.
[YOCTO #3926]
(Bitbake rev: 9ca8e513dc142dd7aefadeb6db2ccdb00f38b39f)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie
Alexandru DAMIAN
Paul Eggleton
Robert Yang
Brendan Le Foll
Bernhard Reutner-Fischer
Namhyung Kim
Robert P. J. Day
Marius Avram
Irina Patru
Jason Wessel
Nathan Rossi
Saul Wold
Cristiana Voicu
Valentin Popa
Andrei Dinu
Christopher Larson