The toaster.sqlite database was located in TOASTER_DIR/build. This
meant that if you named your build directory something else (like cow),
Toaster would fail to make/find the database. TOASTER_DIR is on the
whitelist unlike BUILDDIR and we need to be able to write there anyway
given our current layout so this should not disrupt anything.
[YOCTO #9992]
(Bitbake rev: 2c48168a86309c0cf2be793e7409a78ba21fca14)
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add an example settings that can be used for the basis of the production
instance of Toaster.
[YOCTO #10581]
(Bitbake rev: 1a7c356491b88c8decced39fb2039ef90065f2d2)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As of Django 1.8.16, Django is rejecting any HTTP_HOST header that is
not on the ALLOWED_HOST list. We often need to reference the
toaster server via a fqdn, if we start it via webport=0.0.0.0:8000 for
instance, and are hitting the server from a laptop. This change does
reduce the protection from a DNS rebinding attack, however, if you are
running the toaster server outside a protected network, you should be
using the production instance.
[YOCTO #10578]
(Bitbake rev: 7f51149453c96a3f1da64ea85306518fd2b65f21)
Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use underscore instead of dash in the file name for the test settings.
Also fix the import of the settings module.
(Bitbake rev: 0bdfcafdd1e2ebc10dc0cd343c8bb77f09a71c90)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Used 'rb' mode to open files to avoid unicode error when code
runs on python 3:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80
in position 44: invalid start byte
[YOCTO #9584]
(Bitbake rev: 1414866b84fe1fd674ea79500cd62eda3aa30b33)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When running certain tests we want a particular database specified.
When bitbake toaster ui is being tested pass it these test settings so
that it uses the same database as the unit tests running.
(Bitbake rev: c52e34cac4362ba0a3cb3ea1fcb639e1d802aa85)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We don't need to pass the DATABASE_URL around and read it back if we
setup the django framework in the correct way.
We make the default sqlite database path a full path so that the
database isn't being assumed to be in CWD.
Also add some more useful comments on the database settings.
This is preparation work to migrate the build tests and be able to
trigger builds on differently configured databases.
(Bitbake rev: 973c740404ca6a09feea250d3433075995067fe0)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Dictionary method has_key is deprecated in python 2 and absent
in python 3.
Used '<key> in <dict>' statement to make the code working on
both python 2 and python 3.
[YOCTO #9584]
(Bitbake rev: 3d7ad7ba0d1a6f688ae885817c049f2a8ced11b5)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This updates the print "" syntax to print() and fixes some exception
handling syntax such that its compatible with python v2 and v3.
(Bitbake rev: 58304fcce9727fd89564436771356c033ecd22a3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
python deprecated logger.warn() in favour of logger.warning(). This is only
used in bitbake code so we may as well just translate everything to avoid
warnings under python 3. Its safe for python 2.7.
(Bitbake rev: 676a5f592e8507e81b8f748d58acfea7572f8796)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The error and warning counts displayed for builds were counts of
the errors and warnings objects associated with a build. Because these
values were being derived on the fly, it was not possible to sort by
them.
Previously, the 3rd party django-aggregate-if library was used to
add aggregate fields to Build objects and should then have been
used to populate the "all builds" and "project builds" tables. However,
at some point the templates had changed so that the error and warning
counts were coming from the properties on the Build model and not from
these aggregates. This meant that it was not possible to sort by these
fields.
Django 1.8 supports conditional aggregates in annotation fields on
querysets. This means we can remove django-aggregate-if, use the new Django
1.8 feature to derive errors_no and warnings_no fields as annotations,
then use those annotation fields in the templates. This makes the "builds"
tables sortable again.
[YOCTO #8738]
(Bitbake rev: 9be7c5c18b325f6ed40bc431ac408db242007eb1)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make the default toaster cache dir unique to the user running
toaster. If we have multiple users running toaster we previously
got a permission denied exception on saving a cache file.
[YOCTO #8782]
(Bitbake rev: 5207abdf58019271bf92bff4bcce3911b8691508)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Implemented new management command to check if it's
possible to listen on specified address:port.
[YOCTO #8775]
(Bitbake rev: 0339b90842fd7c878c511b4b89ebcaee9a431bba)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The database URL detection doesn't admit a MySQL URL without a port.
As this is a common case (e.g. you would set the HOST to a mysql.sock
path if pointing at a local MySQL server, with no port setting),
amend the regex so it will correctly recognise paths, as well
as HOST:PORT URLs.
(Bitbake rev: 89386aab888f806d5aa4a8083c06566e48d9445b)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The API for RedirectView is due to change in Django 1.9, which
means that Toaster generates deprecation warnings.
Set the "permanent" flag when constructing RedirectView instances
to prevent this warning.
[YOCTO #8364]
(Bitbake rev: 4aa09488bfe65cb365356b320cd9865643bb4fe5)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Upgrade Django to long-term support version.
Django now provides its own migration framework, so remove
requirement for South.
[YOCTO #8364]
(Bitbake rev: 648b62654c52116451c6a68a46d7264db3a34d09)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As both modes can be used at the same time we can't have any
difference in UI between modes.
Removed all conditional statements that used BUILD_MODE.
(Bitbake rev: 4ce1559c2558bd0fd278ff02a1a93bec03c4156b)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Implemented management command to call getDATABASE_URL API.
It will be used to get database url from toaster shell script
by running 'manage.py get-dburl'
(Bitbake rev: fabe9c8f14fd7b8ab204a2b610c64ac5b62135ac)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We set a TOASTER_MANAGED env variable in the startup script,
which has a value of "1" if Toaster should run in build mode.
Add a BUILD_MODE variable to settings.py which is True if
TOASTER_MANAGED is set to "1", False otherwise.
Add this to the context for every template, so we can use this
information to conditionally alter the content of pages
according to the mode we're in.
[YOCTO #8514]
(Bitbake rev: f9cf4739fc85c9760ce748323dc7c8de3fa7eaec)
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This feature is currently under heavy development and should be used
with caution.
(Bitbake rev: ffc11b2c6c6bac4643233cc46418b025c94607c8)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch cleans up the multiple delete. It:
1) skips build id's that don't exist rather than giving a traceback.
2) let you pass in the ids as a space separated list
3) fixes the usage to match the space separated list format
[YOCTO #7726]
(Bitbake rev: a065f7e5e9c07dbd71a98e7db1d7f711607716f3)
Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prompted by issues discovered during running pylint on
the toaster sources, this patch fixes:
* missing import in toaster ui
* improper call of function in toaster ui (logger.debug)
* improper function definitions in bbcontroller
* invalid references to objects in bldcontrol.models
* proper initialization of object fields in ToasterTables
Also inhibiting specific pylint errors in files where
the problems are mis-identified.
(Bitbake rev: 1c71955c416fb68455f7f70669aba4202c411807)
Signed-off-by: Alexandru Damian <alexandru.damian@intel.com>
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using sqlite we sometimes see Database Locked exceptions when we
fire off database calls asynchronously from the UI. We need sqlite to
wait a bit longer for the lock to be released.
n.b In production setup we hopefully wouldn't be using sqlite.
docs.djangoproject.com/en/1.6/ref/databases/#database-is-locked-errors
(Bitbake rev: 7f3a4e0330f4de3ba8e092bc1c15a53d8c2be073)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds an url to match the old orm application.
(Bitbake rev: 0a8e740e18333da981b24a76db4c891845e5df78)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The orm application was also the django application to collect the build
information. Splitting this module up into it's functional parts. orm
for the data module and bldcollector for build collection data.
(Bitbake rev: 8ca10764ffd6cfec12cbfeabf240d81213a07845)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch fixes the build deletion on unmigrated databases,
and enhances it to delete multiple builds in a single run.
[YOCTO #7726]
(Bitbake rev: d5468d84c1ef83c780de5974c8e3a11eab762489)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We bring the django-aggregate-if module (with minor changes to import)
into the project.
(Bitbake rev: 4b92add34167304b45c66b9726cbc64bc0f74d9f)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to make sure we don't use undefined variables in the
templates, we enforce strict variable checking in the templating
engine.
(Bitbake rev: 3928dc93188880386588fe3f440cd0aaa83d22de)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We remove the endpoints for XHR on the toastergui application.
The endpoints are now replaced with calls to the respective
REST endpoints (i.e. projectlayers, projecttargets, projectmachines).
(Bitbake rev: 8e7a2c3b125a34fd9d6fa0442ab13290137ecc51)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In an effort to fix client-side session storage using
cookies, we enable server-side session support for all pages.
(Bitbake rev: ba10b6f89767c0dad8a2b064f42a84956764e9da)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The SimpleUI is obsolete and not maintained. It should be deleted as
there is no use to it.
[YOCTO #7709]
(Bitbake rev: 10b7c359613629bf6e3465234512990ba4742c48)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When debugging is turned on, we show some Toaster version
information in the top bar. Present it so that it can
be easily copied and pasted.
(Bitbake rev: 5de46fabe9ab1e525048353c2ecd8846ffefc760)
Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch brings in the toaster tree commit ID in
the version string to allow users properly report their
toaster version when submitting issues.
(Bitbake rev: 5cc87ee0ac3b2bd399e7f009dda9d6de87209a27)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch brings in cooker log saving and proper download links.
* toasterui will now write the cooker log file if running in managed
mode
* the BuildRequest has a new state, REQ_ARCHIVE, indicating that the
build is completed, and the artifacts are ready to be grabbed
* the runbuild test execution commands will gather needed artifacts,
and save them to a storage directory selected during Toaster setup.
* the build dashboard, project builds and all builds pages have
permanent links for the cooker log
[YOCTO #7220]
[YOCTO #7206]
(Bitbake rev: fad80e36c9da663b000cdf2cb3c75440c6431d84)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch displays the current running mode and checked-out git branch
as Toaster version when running in debug mode.
(Bitbake rev: 93e4f8c44273f4657c5be4c00b61db12aa875e31)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch improves the logging facilities for toaster in order
to help diagnose bugs that happen on user machines.
The logs are stored now under "/tmp/toaster_$$" where $$ is a
PID-based unique identifier. On shutdown, toaster will automatically
erase all logs unless errors are listed in the log file.
On error, Toaster provides suggestions on what to do.
This patch includes a minor fix found as a result of logging
improvements.
(Bitbake rev: 8a8248f7b7e30469f592e2f8adbf6ce21e8685c5)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch clearers and bring fixes for the layer selection
logic in order to enable information collected during build to be used
in configuring projects, specifically targeting the recipes
learned through the building process.
The patch also adds tests to verify the layer selection logic.
[YOCTO #7189]
(Bitbake rev: f0faba8ef0f08c98ac4bddf5b3954d540820d215)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch enables the server-side caches for Django using
file cache in /tmp/ directory.
Patch enables django debug panel if available.
(Bitbake rev: 00496fb67fa341477edbf80c5438f0b069871ac6)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We update and add logs throughout the code in order to help
with development. The extra logging is turned off by default,
but it can be enabled by using environment variables.
All logging happens through the Python logging facilities.
The toaster UI will save a log of all incoming events if the
TOASTER_EVENTLOG variable is set.
If TOASTER_SQLDEBUG is set all DB queries will be logged.
If TOASTER_DEVEL is set and the django-fresh module is available,
the module is enabled to allow auto-reload of pages when the
source is changed.
(Bitbake rev: 10c27450601b4d24bbb273bd0e053498807d1060)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we are using a virtualenv in the current search path we end up trying
to add modules to the django apps list which do not have the correct
load path or are internal to other modules. Fix this by skipping the
virtualenv path.
[YOCTO #6896]
(Bitbake rev: 3f614295d688c33b690f63ff7eb28e6988707919)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add a logger, enabled through the settings.py, that
logs all SQL queries, with timestamp and duration, to console.
This is helpful in profiling the SQL queries.
The facility is disabled by default.
(Bitbake rev: 54fd0a76fa3154adfab5688ecc96963f42cded11)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We enable the admin interface in Toaster.
We add admin models for Build Environments (where the
sysadmin can configure where the builds take place)
and for Layer Sources (marking the upstream provider
for layer information).
The admin interface and associated data are enabled
only for the MANAGED version.
(Bitbake rev: 6618613c9210fb44d36d90f5f2404b435f10dfc8)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add a ToasterSettings table that will keep
installation-wide settings.
We update the models for the layer-related data storage to
make them compatible with the layerindex application API.
We add a LayerSource class that can update local data from
a LayerIndex-like compatible API.
Adding a command line option to perform information update
from all upstream layer sources.
Fair warning - there is no backward migration from 0013.
(Bitbake rev: 89e13579e1b44b738f10fadec8454aa0e6f073af)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We fix application discovery, tuples don't have .append().
Other minor fixes.
(Bitbake rev: a6f18aac3e6bb448d89a3425a2f756c6514ee595)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add the new project and project page skeletons.
In the process, we add an identifier in the settings.py
to detect whenever Toaster is running in managed mode,
and a context processor to make this value available
to the template processor.
(Bitbake rev: 927a27c68e24cfe13f62ca5f0e60878b04fa4e24)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We automatically enable local applications in INSTALLED_APPS
based on detecting a models.py or views.py file.
This allows Toaster extensibility by adding applications,
without having to edit configuration files.
(Bitbake rev: 05b246095d161143ca2ebc126d9e70d198ab8bb8)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Added code to automatically discover and add url dispatchers
to the urlpattern list. This allows extension of Toaster
through adding applications that will be automatically
registered with the URL dispatcher.
(Bitbake rev: 4000d8224206f404d174a5bc4aa49af55cb44d27)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is implemented as a django management command.
For the moment the 'manage.py perf' command will track the toaster
'gui' urls http response code and load time.
To use it:
1. do your toaster builds
2. ensure toaster is started
1. cd bitbake/lib/toaster
2. ln -s ../../../build/toaster.sqlite
3. ./manage.py perf
(Bitbake rev: 4a1fc6851d21500150715f0e8fa03c0b228ec5f2)
Signed-off-by: Ionut Chisanovici <ionutx.chisanovici@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add a function that returns the DATABASE_URL
for the current 'default' database settings. This
is useful to be able to start other toaster instances
with the same database settigns as the currently running
instance.
(Bitbake rev: 272a4bba0804bb6b5e0d498d3453321b5ed1dc76)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
brian avery
Michael Wood
Ed Bartosh
Richard Purdie
Elliot Smith
brian avery
Alexandru Damian
Belen Barros Pena
Ionut Chisanovici