Commit Graph

50 Commits

Author SHA1 Message Date
Richard Purdie 6f6f51783a bitbake: prserv/persist_data/utils: Drop obsolete python2 imports
These imports were from python 2.6 and earlier, 2.4 in some cases.
Drop them since we're all python3 now.

(Bitbake rev: 7ef12684e8647b006bf46cae695069d4bfece1cf)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-09 13:39:13 +00:00
Richard Purdie 9f6a1043f6 bitbake: prserv/serv: Tweak stdout manipulation to be stream safe
We've been seeing oe-selftest failures under puzzling circumstances. It
turns out if you run oe-selftest on a machine with xmlrunner installed
and have the recent tinfoil2 changes, the launching of PR server crashes
leading to selftest hanging if using an autoloaded PR server.

The reason is that xmlrunner uses an io.StringIO object as stdout/stderr
instead of the usual io.TextIOWrapper and StringIO lacks a fileno() method.

We have to deal with both cases and in the python way, we try and then seek
forgivness if we see an AttributeError or UnSupportedOperation exception.
Unfortunately we have to deal with both cases as we may be performing a
traditiional double fork() from the commandline, or a larger python program.

[YOCTO #10866]

(Bitbake rev: 26243f04e3af652291d13e85c084057104fe155b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-05 13:54:07 +00:00
Joshua Lock 1fce7ecbbb bitbake: bitbake: remove True option to getVar calls
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.

Search made with the following regex: getVar ?\(( ?[^,()]*), True\)

(Bitbake rev: 3b45c479de8640f92dd1d9f147b02e1eecfaadc8)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:09 +00:00
Richard Purdie 0f2c59367a bitbake: bitbake: Convert to python 3
Various misc changes to convert bitbake to python3 which don't warrant
separation into separate commits.

(Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:02 +01:00
Richard Purdie fce8da957f bitbake: daemonize/prserv/tests/fetch: Convert file() -> open()
Use python3 compatible functions.

(Bitbake rev: e6a0296ba29c3fbc8417d1df7a01d50562668a41)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-13 13:41:32 +01:00
Haris Okanovic 4a12865c7b bitbake: prserv: Add dump_db()
Returns a script (string) that reconstructs the state of the
entire database at the time this function is called. The script
language is defined by the backing database engine, which is a
function of server configuration.
Returns None if the database engine does not support dumping to
script or if some other error is encountered in processing.

The SQLite3 implementation in db.py calls iterdump() [1] to generate
a script. iterdump() is the library equivalent of the `sqlite3 .dump`
shell command, and the scripts are compatible. Execute the script in
an empty SQLite3 database using the sqlite3 utility to restore a backup
of prserv.

Use case: Backup a live PR server database in a non-racy way, such
that one could snapshot the entire database after a set of bitbake
builds all using a shared server. I.e. All changes made prior to
the start of a dump_db() operation should be committed and captured
in the script. Subsequent changes made during the backup process are
not guaranteed to be captured.

Testing: ~7MB database backs up in ~1s while PR server is under load
from 32 thread bitbake builds on two separate machines.

[1] https://docs.python.org/2/library/sqlite3.html#sqlite3.Connection.iterdump

(Bitbake rev: 004003daf6bd0f0233ce5c2d95f1d7d64ab91bb3)

Signed-off-by: Haris Okanovic <haris.okanovic@ni.com>
Reviewed-by: Ken Sharp <ken.sharp@ni.com>
Reviewed-by: Bill Pittman <bill.pittman@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-26 17:20:31 +00:00
Diego Santa Cruz 56454f6fb2 bitbake: bitbake: prserv: do not clear umask when daemonizing
Clearing the umask when daemonizing is not the correct thing
to do, as it will create files writable by anyone by default.
For instance the pid file was being created with mode 777.
This could also potentially affect the sqlite database.
Better let the calling process decide on the umask.

[YOCTO #9036]

(Bitbake rev: ff6d3f53a4504eae7ec4c190b9f7595b09aed017)

Signed-off-by: Diego Santa Cruz <Diego.SantaCruz@spinetix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-04 23:47:48 +00:00
Diego Santa Cruz abf8a8fbc3 bitbake: bitbake: prserv: SIGTERM handling hung process
The current SIGTERM handler hungs the process instead of
making it exit. The problem seems to be that the handler thread
is not signaled to quit, so it stays there doing its work, as
it is not a daemon thread. Setting the quit variable fixes this.

While at it, to not use the SystemExit exception to terminate
upon SIGTERM but instead left the quit flag do its job. This way
the PID file is properly removed.

[YOCTO #9035]

(Bitbake rev: 655ec800d54da581229f12efb6f0baf54975fed4)

Signed-off-by: Diego Santa Cruz <Diego.SantaCruz@spinetix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-04 23:47:48 +00:00
Diego Santa Cruz be032fc40e bitbake: bitbake: prserv: -wal and -shm sqlite lost when daemonizing
When daemonizing the PR service the -wal and -shm sqlite files were
being deleted, although they should never be. While the daemonized
process keeps the file descriptors open and thus a clean exit does
not loose any data, a power outage would loose all data in the WAL.
Removing these files also breaks sqlite collaboration between
processes and furthermore prevents taking proper backups without
stopping the PR service.

The reason this happens is that the DB connection is opened in
the initial process, before forking for daemonization. When the
DB connection is closed by the exiting parent processes it can
delete the -wal and -shm files if it considers itself to be the
last connection to the DB. This is easily fixed by opening the
DB connection after all forking.

[YOCTO #9034]

(Bitbake rev: bc867c81e3894da5bdd2e45fa695bb5f5f1bb58b)

Signed-off-by: Diego Santa Cruz <Diego.SantaCruz@spinetix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-04 23:47:48 +00:00
Richard Purdie 5375e6431c bitbake: bitbake: Set process names to be meaninful
This means that when you view the process tree, the processes
have meaningful names, aiding debugging:

$ pstree -p 30021
bash(30021)───KnottyUI(115579)───Cooker(115590)─┬─PRServ(115592)───{PRServ Handler}(115593)
                                                ├─Worker(115630)───bash:sleep(115631)───run.do_sleep.11(115633)───sleep(115634)
                                                └─{ProcessEQueue}(115591)

$ pstree -p 30021
bash(30021)───KnottyUI(117319)───Cooker(117330)─┬─Cooker(117335)
                                                ├─PRServ(117332)───{PRServ Handler}(117333)
                                                ├─Parser-1:2(117336)
                                                └─{ProcessEQueue}(117331)

Applies to parse threads, PR Server, cooker, the workers and execution
threads, working within the 16 character limit as best we can.

Needed to tweak the bitbake-worker magic values to tell the
workers apart.

(Bitbake rev: 539726a3b2202249a3f148d99e08909cb61902a5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-30 11:43:56 +00:00
Leonardo Sandoval f78f90240a bitbake: prserv/serv.py: Better messaging when starting/stopping the server with port=0
When starting the server using port=0, the server actually starts with a
different port, so print a message with this new value. When stopping the
server with port=0, advise the user which ports the server is listening to,
so next time it tries to close it, user can pick up the correct one.

[YOCTO #8560]

(Bitbake rev: 851e53a216682fc9133f51c05a24527cfc677741)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-28 12:00:33 +01:00
Leonardo Sandoval 134b267ebb bitbake: prserv/serv: Close the DB connection out of class destructor
When launching the PR server daemon, the PRData __del__ function was being
called (no reason found yet) where the DB connection closed, thus following
PR updates were not getting into the DB. This patch closes the connection
explicitly, not relaying on the __del__ function execution.
Closing the connection in turn causes all WAL file transactions to be moved
into the database (checkpoint), thus effectively updating the database.

[YOCTO #8215]

(Bitbake rev: c1b4754f69003df1a83fafc1c80a9ef74400b6dd)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-28 12:00:32 +01:00
Leonardo Sandoval 5a51fb28db bitbake: prserv/serv: Start/Stop daemon using ip instead of host
In cases where hostname is given instead of an IP (i.e. localhost
instead of 127.0.0.1) when stopping the server with bitbake-prserv --stop,
the server shows a misleading message indicating that the daemon was not
found, where it is actually stopped. This patch converts host to IP values
before starting/stopping the daemon, so it will always work on IP, not on
hostnames, avoiding problems like the latter.

[YOCTO #8258]

(Bitbake rev: bd6398e967c234e89d773f509512ebf460fa76ff)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-24 17:55:05 +01:00
Richard Purdie 85256c8fa9 bitbake: prserv: SIGTERM and deamonization fixes
Worryingly, if you SIGKILL the bitbake cooker, an autostarted PR server
will remain behind. It turns out there are a few things we should do:

* The PR service doesn't need to daemonize when started from cooker,
  it just complicated the process lifecycle. Add a fork() method
  to handle this and use the non-daemon mode for the singleton.

* Reset the sigterm and sigint handlers. Bitbake cooker installs its
  own which we inherit meaning PR server was ignoring SIGTERM. Installing
  our own handlers which include a sync makes most sense here. Since
  we're in the code, make it sync the database on SIGINT.

* Use the new bb.utils.signal_on_parent_exit() call so that we get a
  SIGTERM when the parent (usually cooker) exits and we can shutdown
  too. Alternatives would be having an open pipe or polling
  os.getppid() for changes but this seems more effective.

(Bitbake rev: 05d31fa1f56bd3d3d363a16a421d9ba7541d4293)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-09 14:27:56 +01:00
Richard Purdie 625a6bf6e8 bitbake: prserv/db: Document history modes
I keep having to dig into the archives to remember this information.
Add it as a comment to the file instead.

(Bitbake rev: 21dce82056887d8d28edde61b1b82f78bdf7613c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-07-12 22:50:40 +01:00
Maxin B. John 29c7111362 bitbake: prserv: serv.py: remove unused and duplicate imports
Remove unused xmlrpclib, atexit and duplicated threading module imports

(Bitbake rev: 3e12f4e8e3ec66d1df772a64be04b90ec72462ae)

Signed-off-by: Maxin B. John <maxin.john@enea.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-05-08 17:46:36 +01:00
Richard Purdie 8da8a0e55e bitbake: prserv/serv: Improve exit handling
Currently, I'm not sure how the prserver managed to shut down cleanly. These
issues may explain some of the hangs people have reported.

This change:

* Ensures the connection acceptance thread monitors self.quit
* We wait for the thread to exit before exitting
* We sync the database when the thread exits
* We do what the comment mentions, timeout after 30s and sync the database
  if needed. Previously, there was no timeout (the 0.5 applies to sockets,
  not the Queue object)

(Bitbake rev: 0926492295d485813d8a4f6b77c7b152e4c5b4c4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-01-23 11:31:58 +00:00
Richard Purdie 7105d44d87 bitbake: prserv: Use WAL mode
Ideally, we want the PR service to have minimal influence from
queued disk IO. sqlite tends to be paranoid about data loss and
locks/fsync calls. There is a "WAL mode" which changes the journalling
mechanism and would appear much better suited to our use case.

This patch therefore switches the database to use WAL mode. With this
change, write overhead appears significantly reduced.

(Bitbake rev: 0cdd48261daeb17efc528b5de0ac81c8836e8565)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-11-06 16:45:23 +00:00
Richard Purdie 2380ebecd1 bitbake: prserv/serv: Ensure sync happens in the correct thread
The sync/commit calls are happening in the submission thread which can
race against the handler. The handler may start new transactions which
then causes the submission thread to error with "cannot start a
transaction within a transaction".

The fix is to move the calls to the correct thread.

(Bitbake rev: 92e128a0e331e563cfe48827e95939041c16c88e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-11-04 15:00:19 +00:00
Ben Shelton cf82c6756b bitbake: prserv: don't wait until exit to sync
In the commit 'prserv: Ensure data is committed', the PR server moved to
only committing transactions to the database when the PR server is
stopped.  This improves performance, but it means that if the machine
running the PR server loses power unexpectedly or if the PR server
process gets SIGKILL, the uncommitted package revision data is lost.

To fix this issue, sync the database periodically, once per 30 seconds
by default, if it has been marked as dirty.  To be safe, continue to
sync the database at exit regardless of its status.

(Bitbake rev: 424df64f2e9679043f0ce2b4f7dfc59c3d404304)

Signed-off-by: Ben Shelton <ben.shelton@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-10-28 15:00:39 +00:00
Konrad Scherer e92e8009a1 bitbake: prserv/serv: Improve error message when prserver cannot bind to supplied host address
If localhost resolves to a remote address (due to a misconfigured network),
starting the pr server will fail without useful information.

To reproduce, add '<bogus ip> localhost' to /etc/hosts and run
'bitbake -p'. The error message will be:

ERROR: Timeout while attempting to communicate with bitbake server
ERROR: Could not connect to server False:

Running 'bitbake-prserv --host=localhost --port=0 --start' will fail with:

error: [Errno 99] Cannot assign requested address

Since these errors does not show the IP address of the attempted socket
binding, this results in a lot of wasted time looking at firewall rules, etc.

This patch results in the following error message if the socket binding fails:

PR Server unable to bind to <bogus ip>:0

(Bitbake rev: fae5914030bcf4c061c22fc61034c40c87b7121a)

Signed-off-by: Konrad Scherer <Konrad.Scherer@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-09-29 12:08:48 +01:00
Robert Yang 053857479c bitbake: lib: fix no newline at end of file
Add a '\n' to the last line of the file to fix:

No newline at end of file

(Bitbake rev: 54f1359ed2e9d47980cd221b7b43ef56543fe06d)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-07-04 09:28:36 +01:00
Richard Purdie 0304d92a97 bitbake: prserv/db: Avoid fsync() calls
If the power were to fail, it doesn't matter to us much if the data
makes it to disk or not, we'd have other problems. However an fsync()
call on a multi build autobuilder is painful so lets avoid them.

(Bitbake rev: 4eb2dc8048e2722d64d589f453df1ce6262c71b8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-05-03 11:24:48 +01:00
Richard Purdie 6448634205 bitbake: prserv: Fix exit race issues
We shouldn't immediately remove the pid file when stopping the server, if we do, this
causes a traceback within the server itself which can then hang. Fix this by removing
the stale pid file as the last thing we do.

Also:

* don't printing a new "waiting" line every 0.5 seconds.
* make the loop more granular since the user can 'feel' the 0.5 seconds

[YOCTO #5984]

(Bitbake rev: 81f41a806aeddcc38992163557672e296bcbc967)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-03-30 10:10:36 +01:00
Konrad Scherer e15893adf9 bitbake: serv.py: Give pr-server up to 5 seconds to commit data
The default value of 0.5 seconds before sending the pr-server a
SIGTERM is not enough to guarantee that sqlite has committed all
the pr data to the database. By polling the pid to see if it is
still running, this allows the pr-server process to shutdown
cleanly and finish the final pr data commit.

(Bitbake rev: 22eec978e70794923c85689928c6be0cfe71cdcd)

Signed-off-by: Konrad Scherer <Konrad.Scherer@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-11-18 17:19:11 +00:00
Richard Purdie 2ab1bf27f7 bitbake: prserv: Ensure data is committed
In exclusive mode, we need to complete the transaction for writes to make
it to the database. Therefore add sync calls to ensure this happens.

Autocommit mode is significantly (100 times) slower so caching the
data is of significant benefit.

(Bitbake rev: 4e55f7821786a59c2cd7dbd8bfa2a22f5f196e99)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-09-08 10:51:56 +01:00
Richard Purdie c7b3429032 bitbake: prserv/serv: Settle on two threads for optimal performance
Using the threading mixin class resulted in large amounts of memory
being used by the PR server for no good reason. Using a receiver thread
and a thread to do the actual database operations on a single connection
gives the same performance with a much saner memory overhead so
switch to this.

(Bitbake rev: e08455d5f3b8e96765942b9c3b9767c30650557d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-09-01 15:51:11 +01:00
Richard Purdie bd023ef9c0 bitbake: serv/db: Don't use BEGIN/COMMIT
Since we don't support using multiple servers on the same database file,
don't use the BEGIN/COMMIT syntax and allow writes to the database
to work ~100 times faster with no transaction locking.

(Bitbake rev: 42144a54979658f93fbbb43f7e271c1fff4d88ff)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-09-01 15:51:11 +01:00
Richard Purdie 883813deae bitbake: serv/db: Take an excluside lock on the database
We only support one server using the database at a time so take an exclusive
lock and avoid later lock overhead.

(Bitbake rev: e3e39be6f2d063858c92971ce8ccd89c95d4f26d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-09-01 15:51:11 +01:00
Richard Purdie 3e5abff7da bitbake: serv/db: Fix looping upon database locked issues
If the database is locked we will get an immediate error indicating so,
there is no retry timeout. The looping code is therefore useless, the loop
count is near instantly exceeded.

Using a time based retry means we can wait a sensible time, then gracefully
exit.

(Bitbake rev: 9f9e6d87007ea87e62495705464f4232c996a165)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-09-01 15:51:11 +01:00
Richard Purdie 5fd04b0258 bitbake: prserv: Allow 'table is locked' matching for retry loop
Try and avoid errors like "ERROR: database table is locked: PRMAIN_nohist"
by retrying if we see the string "is locked".

(Bitbake rev: 1a175b51f80d13f747b653d29e9c0d2201b5109c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-08-30 17:53:26 +01:00
Richard Purdie d5e860be4c bitbake: prserv/serv: Multithread the server
This makes the PR server multithreaded and able to handle multiple connections
at once which means its no longer a build bottle neck when serving one connection
at a time. I've experimented and database connection for each thread seems to
cause the least issues, pushing the contention for sqllite to handle itself.

This means moving the db/table connection code into the actual function methods.
It doesn't abstract well as a function since we need the db object around for
the lifetime of the function as well as the table else we lose the connection.

(Bitbake rev: bf9be2029b2bded5f532bdda4c38ae3dff5d1cf6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-08-29 00:13:22 +01:00
Richard Purdie 06d7664590 bitbake: prserv/db: Threading fixes
Enabling threading for the PRServer causes a number of issues. Firstly is
the obtuse error:

sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type

which is due to the class not being derived from object. See:
http://docs.python.org/2/library/sqlite3.html#registering-an-adapter-callable

Secondly, we want to enable multithreadded access to the database so we do this
when we open it. This opens the way up to multithreading the PR server.

(Bitbake rev: 5709efc2ff1e36529bd28f49cd093ccfa7abff7f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-08-29 00:13:22 +01:00
Jason Wessel 84708a4524 bitbake: serv.py: Fix regression from 972bc43e6d5b
commit 972bc43e6d5b1207b944b3baa8f9805adb35dda7 (serv.py: Fix hang
when spawned dynamically with bitbake) introduced a regression,
because the wrong patch was submitted.  The syntax was incorrect in
the original patch.  The logger iterator must be used with a call to
getLogger().

[YOCTO #5059]

(Bitbake rev: 85fed8acc3af3e15bf119db2f51c486a9de3646b)

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-08-28 09:29:46 +01:00
Jason Wessel da6260f95f bitbake: serv.py: Fix hang when spawned dynamically with bitbake
The PRServer has the possibility to hang indefinitely blocking on a
semaphore processing a xmlrpc request to send an event back to the
main bitbake instance.  This was observed during a "bitbake -e" on a
heavily loaded machine and the main bitbake instance and cooker exited
before the PRServer emitted its first log.

The stack trace is provided below as to show what happens every time a
logger.info() is executed in the PRServer.  Not only does it write to
the stream handler but it also tries to send the event to the main
event processor.

    self._notempty.acquire()
    self.queue.put(event)
    _ui_handlers[h].event.send(event)
    fire_ui_handlers(event, d)
    fire(record, None)
    self.emit(record)
    hdlr.handle(record)
    self.callHandlers(record)
    self.handle(record)
    self._log(INFO, msg, args, **kwargs)
    (self.dbfile, self.host, self.port, str(os.getpid())))
    self.work_forever()
    pid = self.daemonize()
    self.prserv.start()
    singleton.start()
    self.prhost = prserv.serv.auto_start(self.data)
    cooker.pre_serve()
    bb.cooker.server_main(self.cooker, self.main)
    self.run()
    code = process_obj._bootstrap()
    self._popen = Popen(self)
    self.serverImpl.start()
    server.detach()
    server = start_server(servermodule, configParams, configuration)
    ret = main()

It was never intended for the PRServer to send its logs anywhere but
its own log file.  The event processing is an artifact of how the
PRServer was forked and it inherits the event log handlers.  The
simple fix is to clean up and purge all the log handlers after the
fork() but before doing any of the typical PRServer work or logging.

(Bitbake rev: 972bc43e6d5b1207b944b3baa8f9805adb35dda7)

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-08-28 00:25:56 +01:00
Richard Purdie bfab986ccd bitbake: server/xmlrpc/prserv: Add sane timeout to default xmlrpc server
The standard python socket connect has long timouts which make sense for remote
connections but not local things like the PR Service. This adds a timeout
parameter to the common xmlrpc server creation function and sets it to a more
reasonable 5 seconds.

Making the PR server instantly exit is a good way to test the effect of this
on bitbake.

We can remove the bodged timeout in the PRServer terminate function which
has the side effect of affecting global scope.

(Bitbake rev: 8c01cff94787abbb64fbdf0c16cd63f8f97a7e03)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-08-26 11:29:46 +01:00
Richard Purdie 501e1a321d bitbake: prserv/serv: Fix pid file removal
Mark Hatle spotted there were pid files being left around. This patch
fixes things so the removal function is called correctly, the code
contained a typo.

(Bitbake rev: c696a16c8200c31c52750037eeafe07e065b6517)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-08-23 17:43:40 +01:00
Richard Purdie 5ebd9bfff1 bitbake: prserv: Adapt autostart to bitbake-worker
With the change to bitbake-worker we need to ensure the workers know
how to contact the PR service, the magic 0 port and singleton is
no longer enough.

(Bitbake rev: c761751e259bb8e940552a28794b45887b5a72d9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-06-14 12:52:57 +01:00
Richard Purdie 9da832146b bitbake: prserv: Unbreak after bb.server changes
(Bitbake rev: e2cc22fb8b2e97b068b6037540c746ecb1856de6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-05-30 18:05:57 +01:00
Richard Purdie a823c88839 bitbake: prserv: Drop StandardError usage
StandardError doesn't exist in python 3, use Exception instead.

(Bitbake rev: 4a40046036493f0cdf0f66487ad5ce083461a5c2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-05-09 22:28:24 +01:00
Richard Purdie a4cc40c622 bitbake: prserv/cooker: Handle PRService errors cleanly
Current if the PR Service fails to start, bitbake carries on regardless or
hangs with no error message. This adds an exception and then handles it correctly
so the UIs correctly handle the error and exit cleanly.

[YOCTO #4010]

(Bitbake rev: 949c01228a977c3b92bfc0802f6c71b40d8e05b3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-05-03 16:24:13 +01:00
Richard Purdie 1ef926cf78 bitbake: prserv/serv.py: Fix logging in daemon mode
In deamon mode we need to ensure the logging module is sending log data to the
log file. These changes ensure this happens correctly.

(Bitbake rev: bb53b47710ca4579e20284668cb354f734c3d502)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-02-06 23:45:36 +00:00
Richard Purdie bdfc5207a0 bitbake: bitbake: Always use separate process for PR Service
Using the threading module interacts badly with multiprocessing used elsewhere
in bitbake under certain machine loads. This was leading to bitbake hanging on
Ctrl+C when the PR Server was being used.

This patch converts it to always use the daemonize code which
then means the threading code isn't required.

[YOCTO #3742]

(Bitbake rev: 2d0bbd9398ab839bd2d1e29e50b25d52efb1ce2a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-02-06 23:45:36 +00:00
Constantin Musca 1db3531cd5 bitbake: prserv: use only PRSERV_HOST
- remove PRSERV_PORT variable
- use 'hostname:port' as PRSERV_HOST format
- use 'localhost:0' for enabling the local PRServer

[YOCTO #3744]

(Bitbake rev: ad62eed9e1f0867a406e9fbfa44916a0f1ad1282)

Signed-off-by: Constantin Musca <constantinx.musca@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-01-25 12:42:08 +00:00
Lianhao Lu 555262744d prserv: Do not ping PRService if not required
[YOCTO #1942]
Ping the PRService only if PRSERV_HOST and PRSERV_PORT are set.

(Bitbake rev: 20f24de0bdafac21f5d8a58701f977efa7041288)

Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-02-01 15:08:42 +00:00
Martin Jansa 3c5f108001 prserv: fix import of sqlite3
* this is used in all other bitbake parts where sqlite3 is used, don't
  know why it wasn't used here, but it fails e.g. on Gentoo
  Traceback (most recent call last):
    File "bin/bitbake", line 39, in <module>
      from bb import cooker
    File "lib/bb/cooker.py", line 39, in <module>
      import prserv.serv
    File "lib/prserv/serv.py", line 4, in <module>
      import xmlrpclib,sqlite3
  ImportError: No module named sqlite3

(Bitbake rev: 9a57ec705cf5c932d8c2a35852db7a4627c57937)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-01-20 16:47:49 +00:00
Lianhao Lu 489cde8eb0 bitbake: Automatically start local PR service.
[YOCTO #1126]
A local PR service will be started and stopped automatically along
with the bitbake invocation/ternimation.

This local PR service will be started only and if only when the
PRSERV_HOST is set to 'localhost' and PRSERV_PORT is set to '0'.

When started, the sqlite3 database is stored at
"${PERSISTEN_DIR}/prserv.sqlite3" or "${CACHE}/prserv.sqlite3".

(Bitbake rev: 9d8f45407c67ed0d3c4f820cf646de3c385067c7)

Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-01-11 10:36:20 +00:00
Lianhao Lu 30a9bc6c92 bitbake/PRservice: Added no_hist mode and export/import.
[YOCTO #1556]
1. Added the package_arch into the index to the DB table. Because the
change in PACKAGE_ARCH will results in different checksum, and it is
better to have seperate PR value domains for differnt PACKAGE_ARCH of
the same pakcage.

2. Changed the PR service to operate in no history mode. In this mode,
the for a given query tuple (version, pkgarch, checksum), the returned
value will be the largest among all the values of the same (version,
pkgarch). This means the PR value returned can NOT be decremented.

3. Added export function. For each (version, pkgarch) tuple, only the
record with the maximum value will be exported.

4. Added import function. The record will only be imported if the
imported value is larger than the value stored in the DB with the same
(version, pkgarch, checksum) tuple.

(Bitbake rev: 379567ee879dcdc09a51f7f1212bde1076147a6f)

Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-01-11 10:36:20 +00:00
Scott Garman 62d538fbe6 make exception handling syntax consistent
Update exception handling syntax to use the modern style:
except ExcType as localvar

(Bitbake rev: dbf5f42b06bef81749b13aa99945cc1292a6676d)

Signed-off-by: Scott Garman <scott.a.garman@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-06-15 11:13:13 +01:00
Lianhao Lu ecdbd6ab03 Add PR service deamon to bitbake
Added the initial implementation of the server side PR service.

(Bitbake rev: 4d0e79e5591ff58ce35c7fb96f6e9217ddc27466)

Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-05-27 17:55:49 +01:00