bitbake: toaster: Remove contrib tts
Remove the "Toaster test system". We don't need a home brew test "framework" as the django test runner is more than adequate. None of these tests here are currently working and have been obsoleted by the work done on unit and browser tests in ./tests/. (Bitbake rev: 7a82e45ca5c4d470f62f83e72d00cbe45baa1537) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is contained in:
parent
953b2f50f0
commit
5bc6fa01e0
|
@ -1,6 +0,0 @@
|
||||||
contrib directory for toaster
|
|
||||||
|
|
||||||
This directory holds code that works with Toaster, without being an integral part of the Toaster project.
|
|
||||||
It is intended for testing code, testing fixtures, tools for Toaster, etc.
|
|
||||||
|
|
||||||
NOTE: This directory is NOT a Python module.
|
|
|
@ -1,41 +0,0 @@
|
||||||
|
|
||||||
Toaster Testing Framework
|
|
||||||
Yocto Project
|
|
||||||
|
|
||||||
|
|
||||||
Rationale
|
|
||||||
------------
|
|
||||||
As Toaster contributions grow with the number of people that contribute code, verifying each patch prior to submitting upstream becomes a hard-to-scale problem for humans. We devised this system in order to run patch-level validation, trying to eliminate common problems from submitted patches, in an automated fashion.
|
|
||||||
|
|
||||||
The Toaster Testing Framework is a set of Python scripts that provides an extensible way to write smoke and regression tests that will be run on each patch set sent for review on the toaster mailing list.
|
|
||||||
|
|
||||||
|
|
||||||
Usage
|
|
||||||
------------
|
|
||||||
There are three main executable scripts in this directory.
|
|
||||||
* runner.py is designed to be run from the command line. It requires, as mandatory parameter, a branch name on poky-contrib, branch which contains the patches to be tested. The program will auto-discover the available tests residing in this directory by looking for unittest classes, and will run the tests on the branch dumping the output to the standard output. Optionally, it can take parameters inhibiting the branch checkout, or specifying a single test to be run, for debugging purposes.
|
|
||||||
* launcher.py is a designed to be run from a crontab or similar scheduling mechanism. It looks up a backlog file containing branches-to-test (named tasks in the source code), select the first one in FIFO manner, and launch runner.py on it. It will await for completion, and email the standard output and standard error dumps from the runner.py execution
|
|
||||||
* recv.py is an email receiver, designed to be called as a pipe from a .forward file. It is used to monitor a mailing list, for example, and add tasks to the backlog based on review requests coming on the mailing list.
|
|
||||||
|
|
||||||
|
|
||||||
Installation
|
|
||||||
------------
|
|
||||||
As prerequisite, we expect a functioning email system on a machine with Python2.
|
|
||||||
|
|
||||||
The broad steps to installation
|
|
||||||
* set up the .forward on the receiving email account to pipe to the recv.py file
|
|
||||||
* edit config.py and settings.json to alter for local installation settings
|
|
||||||
* on email receive, verify backlog.txt to see that the tasks are received and marked for processing
|
|
||||||
* execute launcher.py in command line to verify that a test occurs with no problems, and that the outgoing email is delivered
|
|
||||||
* add launcher.py
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
Contribute
|
|
||||||
------------
|
|
||||||
What we need are tests. Add your own tests to either tests.py file, or to a new file.
|
|
||||||
Use "config.logger" to write logs that will make it to email.
|
|
||||||
|
|
||||||
Commonly used code should be going to shellutils, and configuration to config.py.
|
|
||||||
|
|
||||||
Contribute code by emailing patches to the list: toaster@yoctoproject.org (membership required)
|
|
|
@ -1,9 +0,0 @@
|
||||||
We need to implement tests:
|
|
||||||
|
|
||||||
automated link checker; currently
|
|
||||||
$ linkchecker -t 1000 -F csv http://localhost:8000/
|
|
||||||
|
|
||||||
integrate the w3c-validation service; currently
|
|
||||||
$ python urlcheck.py
|
|
||||||
|
|
||||||
|
|
|
@ -1,98 +0,0 @@
|
||||||
#!/usr/bin/python
|
|
||||||
|
|
||||||
# ex:ts=4:sw=4:sts=4:et
|
|
||||||
# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
|
|
||||||
#
|
|
||||||
# Copyright (C) 2015 Alexandru Damian for Intel Corp.
|
|
||||||
#
|
|
||||||
# This program is free software; you can redistribute it and/or modify
|
|
||||||
# it under the terms of the GNU General Public License version 2 as
|
|
||||||
# published by the Free Software Foundation.
|
|
||||||
#
|
|
||||||
# This program is distributed in the hope that it will be useful,
|
|
||||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
||||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
||||||
# GNU General Public License for more details.
|
|
||||||
#
|
|
||||||
# You should have received a copy of the GNU General Public License along
|
|
||||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
|
||||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
|
||||||
|
|
||||||
# This is the configuration/single module for tts
|
|
||||||
# everything that would be a global variable goes here
|
|
||||||
|
|
||||||
import os, sys, logging
|
|
||||||
import socket
|
|
||||||
|
|
||||||
LOGDIR = "log"
|
|
||||||
SETTINGS_FILE = os.path.join(os.path.dirname(__file__), "settings.json")
|
|
||||||
TEST_DIR_NAME = "tts_testdir"
|
|
||||||
|
|
||||||
DEBUG = True
|
|
||||||
|
|
||||||
OWN_PID = os.getpid()
|
|
||||||
|
|
||||||
W3C_VALIDATOR = "http://icarus.local/w3c-validator/check?doctype=HTML5&uri="
|
|
||||||
|
|
||||||
TOASTER_PORT = 56789
|
|
||||||
|
|
||||||
TESTDIR = None
|
|
||||||
|
|
||||||
#we parse the w3c URL to know where to connect
|
|
||||||
|
|
||||||
import urlparse
|
|
||||||
|
|
||||||
def get_public_ip():
|
|
||||||
temp_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
|
|
||||||
parsed_url = urlparse.urlparse("http://icarus.local/w3c-validator/check?doctype=HTML5&uri=")
|
|
||||||
temp_socket.connect((parsed_url.netloc, 80 if parsed_url.port is None else parsed_url.port))
|
|
||||||
public_ip = temp_socket.getsockname()[0]
|
|
||||||
temp_socket.close()
|
|
||||||
return public_ip
|
|
||||||
|
|
||||||
TOASTER_BASEURL = "http://%s:%d/" % (get_public_ip(), TOASTER_PORT)
|
|
||||||
|
|
||||||
|
|
||||||
OWN_EMAIL_ADDRESS = "Toaster Testing Framework <alexandru.damian@intel.com>"
|
|
||||||
REPORT_EMAIL_ADDRESS = "alexandru.damian@intel.com"
|
|
||||||
|
|
||||||
# make sure we have the basic logging infrastructure
|
|
||||||
|
|
||||||
#pylint: disable=invalid-name
|
|
||||||
# we disable the invalid name because the module-level "logger" is used througout bitbake
|
|
||||||
logger = logging.getLogger("toastertest")
|
|
||||||
__console__ = logging.StreamHandler(sys.stdout)
|
|
||||||
__console__.setFormatter(logging.Formatter("%(asctime)s %(levelname)s: %(message)s"))
|
|
||||||
logger.addHandler(__console__)
|
|
||||||
logger.setLevel(logging.DEBUG)
|
|
||||||
|
|
||||||
|
|
||||||
# singleton file names
|
|
||||||
LOCKFILE = "/tmp/ttf.lock"
|
|
||||||
BACKLOGFILE = os.path.join(os.path.dirname(__file__), "backlog.txt")
|
|
||||||
|
|
||||||
# task states
|
|
||||||
def enum(*sequential, **named):
|
|
||||||
enums = dict(zip(sequential, range(len(sequential))), **named)
|
|
||||||
reverse = dict((value, key) for key, value in enums.items())
|
|
||||||
enums['reverse_mapping'] = reverse
|
|
||||||
return type('Enum', (), enums)
|
|
||||||
|
|
||||||
|
|
||||||
class TASKS(object):
|
|
||||||
#pylint: disable=too-few-public-methods
|
|
||||||
PENDING = "PENDING"
|
|
||||||
INPROGRESS = "INPROGRESS"
|
|
||||||
DONE = "DONE"
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def next_task(task):
|
|
||||||
if task == TASKS.PENDING:
|
|
||||||
return TASKS.INPROGRESS
|
|
||||||
if task == TASKS.INPROGRESS:
|
|
||||||
return TASKS.DONE
|
|
||||||
raise Exception("Invalid next task state for %s" % task)
|
|
||||||
|
|
||||||
# TTS specific
|
|
||||||
CONTRIB_REPO = "git@git.yoctoproject.org:poky-contrib"
|
|
||||||
|
|
|
@ -1,101 +0,0 @@
|
||||||
#!/usr/bin/python
|
|
||||||
|
|
||||||
# ex:ts=4:sw=4:sts=4:et
|
|
||||||
# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
|
|
||||||
#
|
|
||||||
# Copyright (C) 2015 Alexandru Damian for Intel Corp.
|
|
||||||
#
|
|
||||||
# This program is free software; you can redistribute it and/or modify
|
|
||||||
# it under the terms of the GNU General Public License version 2 as
|
|
||||||
# published by the Free Software Foundation.
|
|
||||||
#
|
|
||||||
# This program is distributed in the hope that it will be useful,
|
|
||||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
||||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
||||||
# GNU General Public License for more details.
|
|
||||||
#
|
|
||||||
# You should have received a copy of the GNU General Public License along
|
|
||||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
|
||||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
|
||||||
|
|
||||||
# Program to run the next task listed from the backlog.txt; designed to be
|
|
||||||
# run from crontab.
|
|
||||||
|
|
||||||
from __future__ import print_function
|
|
||||||
import sys, os, config, shellutils
|
|
||||||
from shellutils import ShellCmdException
|
|
||||||
|
|
||||||
# Import smtplib for the actual sending function
|
|
||||||
import smtplib
|
|
||||||
|
|
||||||
# Import the email modules we'll need
|
|
||||||
from email.mime.text import MIMEText
|
|
||||||
|
|
||||||
def _take_lockfile():
|
|
||||||
return shellutils.lockfile(shellutils.mk_lock_filename())
|
|
||||||
|
|
||||||
|
|
||||||
def read_next_task_by_state(task_state, task_name=None):
|
|
||||||
if not os.path.exists(os.path.join(os.path.dirname(__file__), config.BACKLOGFILE)):
|
|
||||||
return None
|
|
||||||
os.rename(config.BACKLOGFILE, config.BACKLOGFILE + ".tmp")
|
|
||||||
task = None
|
|
||||||
with open(config.BACKLOGFILE + ".tmp", "r") as f_in:
|
|
||||||
with open(config.BACKLOGFILE, "w") as f_out:
|
|
||||||
for line in f_in.readlines():
|
|
||||||
if task is None:
|
|
||||||
fields = line.strip().split("|", 2)
|
|
||||||
if fields[1] == task_state:
|
|
||||||
if task_name is None or task_name == fields[0]:
|
|
||||||
task = fields[0]
|
|
||||||
print("Updating %s %s to %s" % (task, task_state, config.TASKS.next_task(task_state)))
|
|
||||||
line = "%s|%s\n" % (task, config.TASKS.next_task(task_state))
|
|
||||||
f_out.write(line)
|
|
||||||
os.remove(config.BACKLOGFILE + ".tmp")
|
|
||||||
return task
|
|
||||||
|
|
||||||
def send_report(task_name, plaintext, errtext=None):
|
|
||||||
if errtext is None:
|
|
||||||
msg = MIMEText(plaintext)
|
|
||||||
else:
|
|
||||||
if plaintext is None:
|
|
||||||
plaintext = ""
|
|
||||||
msg = MIMEText("--STDOUT dump--\n\n%s\n\n--STDERR dump--\n\n%s" % (plaintext, errtext))
|
|
||||||
|
|
||||||
msg['Subject'] = "[review-request] %s - smoke test results" % task_name
|
|
||||||
msg['From'] = config.OWN_EMAIL_ADDRESS
|
|
||||||
msg['To'] = config.REPORT_EMAIL_ADDRESS
|
|
||||||
|
|
||||||
smtp_connection = smtplib.SMTP("localhost")
|
|
||||||
smtp_connection.sendmail(config.OWN_EMAIL_ADDRESS, [config.REPORT_EMAIL_ADDRESS], msg.as_string())
|
|
||||||
smtp_connection.quit()
|
|
||||||
|
|
||||||
def main():
|
|
||||||
# we don't do anything if we have another instance of us running
|
|
||||||
lock_file = _take_lockfile()
|
|
||||||
|
|
||||||
if lock_file is None:
|
|
||||||
if config.DEBUG:
|
|
||||||
print("Concurrent script in progress, exiting")
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
next_task = read_next_task_by_state(config.TASKS.PENDING)
|
|
||||||
if next_task is not None:
|
|
||||||
print("Next task is", next_task)
|
|
||||||
errtext = None
|
|
||||||
out = None
|
|
||||||
try:
|
|
||||||
out = shellutils.run_shell_cmd("%s %s" % (os.path.join(os.path.dirname(__file__), "runner.py"), next_task))
|
|
||||||
except ShellCmdException as exc:
|
|
||||||
print("Failed while running the test runner: %s", exc)
|
|
||||||
errtext = exc.__str__()
|
|
||||||
send_report(next_task, out, errtext)
|
|
||||||
read_next_task_by_state(config.TASKS.INPROGRESS, next_task)
|
|
||||||
else:
|
|
||||||
print("No task")
|
|
||||||
|
|
||||||
shellutils.unlockfile(lock_file)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
|
@ -1,56 +0,0 @@
|
||||||
#!/usr/bin/python
|
|
||||||
|
|
||||||
# ex:ts=4:sw=4:sts=4:et
|
|
||||||
# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
|
|
||||||
#
|
|
||||||
# Copyright (C) 2015 Alexandru Damian for Intel Corp.
|
|
||||||
#
|
|
||||||
# This program is free software; you can redistribute it and/or modify
|
|
||||||
# it under the terms of the GNU General Public License version 2 as
|
|
||||||
# published by the Free Software Foundation.
|
|
||||||
#
|
|
||||||
# This program is distributed in the hope that it will be useful,
|
|
||||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
||||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
||||||
# GNU General Public License for more details.
|
|
||||||
#
|
|
||||||
# You should have received a copy of the GNU General Public License along
|
|
||||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
|
||||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
|
||||||
|
|
||||||
# Program to receive review requests by email and log tasks to backlog.txt
|
|
||||||
# Designed to be run by the email system from a .forward file:
|
|
||||||
#
|
|
||||||
# cat .forward
|
|
||||||
# |[full/path]/recv.py
|
|
||||||
|
|
||||||
from __future__ import print_function
|
|
||||||
import sys, config, shellutils
|
|
||||||
|
|
||||||
from email.parser import Parser
|
|
||||||
|
|
||||||
def recv_mail(datastring):
|
|
||||||
headers = Parser().parsestr(datastring)
|
|
||||||
return headers['subject']
|
|
||||||
|
|
||||||
def main():
|
|
||||||
lock_file = shellutils.lockfile(shellutils.mk_lock_filename(), retry=True)
|
|
||||||
|
|
||||||
if lock_file is None:
|
|
||||||
if config.DEBUG:
|
|
||||||
print("Concurrent script in progress, exiting")
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
subject = recv_mail(sys.stdin.read())
|
|
||||||
|
|
||||||
subject_parts = subject.split()
|
|
||||||
if "[review-request]" in subject_parts:
|
|
||||||
task_name = subject_parts[subject_parts.index("[review-request]") + 1]
|
|
||||||
with open(config.BACKLOGFILE, "a") as fout:
|
|
||||||
line = "%s|%s\n" % (task_name, config.TASKS.PENDING)
|
|
||||||
fout.write(line)
|
|
||||||
|
|
||||||
shellutils.unlockfile(lock_file)
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
|
@ -1,222 +0,0 @@
|
||||||
#!/usr/bin/python
|
|
||||||
|
|
||||||
# ex:ts=4:sw=4:sts=4:et
|
|
||||||
# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
|
|
||||||
#
|
|
||||||
# Copyright (C) 2015 Alexandru Damian for Intel Corp.
|
|
||||||
#
|
|
||||||
# This program is free software; you can redistribute it and/or modify
|
|
||||||
# it under the terms of the GNU General Public License version 2 as
|
|
||||||
# published by the Free Software Foundation.
|
|
||||||
#
|
|
||||||
# This program is distributed in the hope that it will be useful,
|
|
||||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
||||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
||||||
# GNU General Public License for more details.
|
|
||||||
#
|
|
||||||
# You should have received a copy of the GNU General Public License along
|
|
||||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
|
||||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
|
||||||
|
|
||||||
|
|
||||||
# This is the main test execution controller. It is designed to be run
|
|
||||||
# manually from the command line, or to be called from a different program
|
|
||||||
# that schedules test execution.
|
|
||||||
#
|
|
||||||
# Execute runner.py -h for help.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
from __future__ import print_function
|
|
||||||
import sys, os
|
|
||||||
import unittest, importlib
|
|
||||||
import logging, pprint, json
|
|
||||||
import re
|
|
||||||
from shellutils import ShellCmdException, mkdirhier, run_shell_cmd
|
|
||||||
|
|
||||||
import config
|
|
||||||
|
|
||||||
# we also log to a file, in addition to console, because our output is important
|
|
||||||
__log_file_name__ = os.path.join(os.path.dirname(__file__), "log/tts_%d.log" % config.OWN_PID)
|
|
||||||
mkdirhier(os.path.dirname(__log_file_name__))
|
|
||||||
__log_file__ = open(__log_file_name__, "w")
|
|
||||||
__file_handler__ = logging.StreamHandler(__log_file__)
|
|
||||||
__file_handler__.setFormatter(logging.Formatter("%(asctime)s %(levelname)s: %(message)s"))
|
|
||||||
|
|
||||||
config.logger.addHandler(__file_handler__)
|
|
||||||
|
|
||||||
# set up log directory
|
|
||||||
try:
|
|
||||||
if not os.path.exists(config.LOGDIR):
|
|
||||||
os.mkdir(config.LOGDIR)
|
|
||||||
else:
|
|
||||||
if not os.path.isdir(config.LOGDIR):
|
|
||||||
raise Exception("Expected log dir '%s' is not actually a directory." % config.LOGDIR)
|
|
||||||
except OSError as exc:
|
|
||||||
raise exc
|
|
||||||
|
|
||||||
# creates the under-test-branch as a separate directory
|
|
||||||
def set_up_test_branch(settings, branch_name):
|
|
||||||
testdir = "%s/%s.%d" % (settings['workdir'], config.TEST_DIR_NAME, config.OWN_PID)
|
|
||||||
|
|
||||||
# creates the host dir
|
|
||||||
if os.path.exists(testdir):
|
|
||||||
raise Exception("Test dir '%s'is already there, aborting" % testdir)
|
|
||||||
|
|
||||||
# may raise OSError, is to be handled by the caller
|
|
||||||
os.makedirs(testdir)
|
|
||||||
|
|
||||||
|
|
||||||
# copies over the .git from the localclone
|
|
||||||
run_shell_cmd("cp -a '%s'/.git '%s'" % (settings['localclone'], testdir))
|
|
||||||
|
|
||||||
# add the remote if it doesn't exist
|
|
||||||
crt_remotes = run_shell_cmd("git remote -v", cwd=testdir)
|
|
||||||
remotes = [word for line in crt_remotes.split("\n") for word in line.split()]
|
|
||||||
if not config.CONTRIB_REPO in remotes:
|
|
||||||
remote_name = "tts_contrib"
|
|
||||||
run_shell_cmd("git remote add %s %s" % (remote_name, config.CONTRIB_REPO), cwd=testdir)
|
|
||||||
else:
|
|
||||||
remote_name = remotes[remotes.index(config.CONTRIB_REPO) - 1]
|
|
||||||
|
|
||||||
# do the fetch
|
|
||||||
run_shell_cmd("git fetch %s -p" % remote_name, cwd=testdir)
|
|
||||||
|
|
||||||
# do the checkout
|
|
||||||
run_shell_cmd("git checkout origin/master && git branch -D %s; git checkout %s/%s -b %s && git reset --hard" % (branch_name, remote_name, branch_name, branch_name), cwd=testdir)
|
|
||||||
|
|
||||||
return testdir
|
|
||||||
|
|
||||||
|
|
||||||
def __search_for_tests():
|
|
||||||
# we find all classes that can run, and run them
|
|
||||||
tests = []
|
|
||||||
for _, _, files_list in os.walk(os.path.dirname(os.path.abspath(__file__))):
|
|
||||||
for module_file in [f[:-3] for f in files_list if f.endswith(".py") and not f.startswith("__init__")]:
|
|
||||||
config.logger.debug("Inspecting module %s", module_file)
|
|
||||||
current_module = importlib.import_module(module_file)
|
|
||||||
crtclass_names = vars(current_module)
|
|
||||||
for name in crtclass_names:
|
|
||||||
tested_value = crtclass_names[name]
|
|
||||||
if isinstance(tested_value, type(unittest.TestCase)) and issubclass(tested_value, unittest.TestCase):
|
|
||||||
tests.append((module_file, name))
|
|
||||||
break
|
|
||||||
return tests
|
|
||||||
|
|
||||||
|
|
||||||
# boilerplate to self discover tests and run them
|
|
||||||
def execute_tests(dir_under_test, testname):
|
|
||||||
|
|
||||||
if testname is not None and "." in testname:
|
|
||||||
tests = []
|
|
||||||
tests.append(tuple(testname.split(".", 2)))
|
|
||||||
else:
|
|
||||||
tests = __search_for_tests()
|
|
||||||
|
|
||||||
# let's move to the directory under test
|
|
||||||
crt_dir = os.getcwd()
|
|
||||||
os.chdir(dir_under_test)
|
|
||||||
|
|
||||||
# execute each module
|
|
||||||
# pylint: disable=broad-except
|
|
||||||
# we disable the broad-except because we want to actually catch all possible exceptions
|
|
||||||
try:
|
|
||||||
# sorting the tests by the numeric order in the class name
|
|
||||||
tests = sorted(tests, key=lambda x: int(re.search(r"[0-9]+", x[1]).group(0)))
|
|
||||||
config.logger.debug("Discovered test clases: %s", pprint.pformat(tests))
|
|
||||||
unittest.installHandler()
|
|
||||||
suite = unittest.TestSuite()
|
|
||||||
loader = unittest.TestLoader()
|
|
||||||
result = unittest.TestResult()
|
|
||||||
result.failfast = True
|
|
||||||
for module_file, test_name in tests:
|
|
||||||
suite.addTest(loader.loadTestsFromName("%s.%s" % (module_file, test_name)))
|
|
||||||
config.logger.info("Running %d test(s)", suite.countTestCases())
|
|
||||||
suite.run(result)
|
|
||||||
|
|
||||||
for error in result.errors:
|
|
||||||
config.logger.error("Exception on test: %s\n%s", error[0],
|
|
||||||
"\n".join(["-- %s" % x for x in error[1].split("\n")]))
|
|
||||||
|
|
||||||
for failure in result.failures:
|
|
||||||
config.logger.error("Failed test: %s:\n%s\n", failure[0],
|
|
||||||
"\n".join(["-- %s" % x for x in failure[1].split("\n")]))
|
|
||||||
|
|
||||||
config.logger.info("Test results: %d ran, %d errors, %d failures", result.testsRun, len(result.errors), len(result.failures))
|
|
||||||
|
|
||||||
except Exception as exc:
|
|
||||||
import traceback
|
|
||||||
config.logger.error("Exception while running test. Tracedump: \n%s", traceback.format_exc())
|
|
||||||
finally:
|
|
||||||
os.chdir(crt_dir)
|
|
||||||
return len(result.failures)
|
|
||||||
|
|
||||||
# verify that we had a branch-under-test name as parameter
|
|
||||||
def validate_args():
|
|
||||||
from optparse import OptionParser
|
|
||||||
parser = OptionParser(usage="usage: %prog [options] branch_under_test")
|
|
||||||
|
|
||||||
parser.add_option("-t", "--test-dir", dest="testdir", default=None, help="Use specified directory to run tests, inhibits the checkout.")
|
|
||||||
parser.add_option("-s", "--single", dest="singletest", default=None, help="Run only the specified test")
|
|
||||||
|
|
||||||
(options, args) = parser.parse_args()
|
|
||||||
if len(args) < 1:
|
|
||||||
raise Exception("Please specify the branch to run on. Use option '-h' when in doubt.")
|
|
||||||
return (options, args)
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
# load the configuration options
|
|
||||||
def read_settings():
|
|
||||||
if not os.path.exists(config.SETTINGS_FILE) or not os.path.isfile(config.SETTINGS_FILE):
|
|
||||||
raise Exception("Config file '%s' cannot be openend" % config.SETTINGS_FILE)
|
|
||||||
return json.loads(open(config.SETTINGS_FILE, "r").read())
|
|
||||||
|
|
||||||
|
|
||||||
# cleanup !
|
|
||||||
def clean_up(testdir):
|
|
||||||
run_shell_cmd("rm -rf -- '%s'" % testdir)
|
|
||||||
|
|
||||||
def dump_info(settings, options, args):
|
|
||||||
""" detailed information about current run configuration, for debugging purposes.
|
|
||||||
"""
|
|
||||||
config.logger.debug("Settings:\n%s\nOptions:\n%s\nArguments:\n%s\n", settings, options, args)
|
|
||||||
|
|
||||||
def main():
|
|
||||||
(options, args) = validate_args()
|
|
||||||
|
|
||||||
settings = read_settings()
|
|
||||||
need_cleanup = False
|
|
||||||
|
|
||||||
# dump debug info
|
|
||||||
dump_info(settings, options, args)
|
|
||||||
|
|
||||||
testdir = None
|
|
||||||
no_failures = 1
|
|
||||||
try:
|
|
||||||
if options.testdir is not None and os.path.exists(options.testdir):
|
|
||||||
testdir = os.path.abspath(options.testdir)
|
|
||||||
config.logger.info("No checkout, using %s", testdir)
|
|
||||||
else:
|
|
||||||
need_cleanup = True
|
|
||||||
testdir = set_up_test_branch(settings, args[0]) # we expect a branch name as first argument
|
|
||||||
|
|
||||||
config.TESTDIR = testdir # we let tests know where to run
|
|
||||||
|
|
||||||
# ensure that the test dir only contains no *.pyc leftovers
|
|
||||||
run_shell_cmd("find '%s' -type f -name *.pyc -exec rm {} \\;" % testdir)
|
|
||||||
|
|
||||||
no_failures = execute_tests(testdir, options.singletest)
|
|
||||||
|
|
||||||
except ShellCmdException as exc:
|
|
||||||
import traceback
|
|
||||||
config.logger.error("Error while setting up testing. Traceback: \n%s", traceback.format_exc())
|
|
||||||
finally:
|
|
||||||
if need_cleanup and testdir is not None:
|
|
||||||
clean_up(testdir)
|
|
||||||
|
|
||||||
sys.exit(no_failures)
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
|
@ -1,5 +0,0 @@
|
||||||
{
|
|
||||||
"repo": "git@git.yoctoproject.org:poky-contrib",
|
|
||||||
"localclone": "/home/ddalex/ssd/yocto/poky",
|
|
||||||
"workdir": "/home/ddalex/ssd/yocto"
|
|
||||||
}
|
|
|
@ -1,141 +0,0 @@
|
||||||
#!/usr/bin/python
|
|
||||||
|
|
||||||
# ex:ts=4:sw=4:sts=4:et
|
|
||||||
# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
|
|
||||||
#
|
|
||||||
# Copyright (C) 2015 Alexandru Damian for Intel Corp.
|
|
||||||
#
|
|
||||||
# This program is free software; you can redistribute it and/or modify
|
|
||||||
# it under the terms of the GNU General Public License version 2 as
|
|
||||||
# published by the Free Software Foundation.
|
|
||||||
#
|
|
||||||
# This program is distributed in the hope that it will be useful,
|
|
||||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
||||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
||||||
# GNU General Public License for more details.
|
|
||||||
#
|
|
||||||
# You should have received a copy of the GNU General Public License along
|
|
||||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
|
||||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
|
||||||
|
|
||||||
# Utilities shared by tests and other common bits of code.
|
|
||||||
|
|
||||||
import sys, os, subprocess, fcntl, errno
|
|
||||||
import config
|
|
||||||
from config import logger
|
|
||||||
|
|
||||||
|
|
||||||
# License warning; this code is copied from the BitBake project, file bitbake/lib/bb/utils.py
|
|
||||||
# The code is originally licensed GPL-2.0, and we redistribute it under still GPL-2.0
|
|
||||||
|
|
||||||
# End of copy is marked with #ENDOFCOPY marker
|
|
||||||
|
|
||||||
def mkdirhier(directory):
|
|
||||||
"""Create a directory like 'mkdir -p', but does not complain if
|
|
||||||
directory already exists like os.makedirs
|
|
||||||
"""
|
|
||||||
|
|
||||||
try:
|
|
||||||
os.makedirs(directory)
|
|
||||||
except OSError as exc:
|
|
||||||
if exc.errno != errno.EEXIST:
|
|
||||||
raise exc
|
|
||||||
|
|
||||||
def lockfile(name, shared=False, retry=True):
|
|
||||||
"""
|
|
||||||
Use the file fn as a lock file, return when the lock has been acquired.
|
|
||||||
Returns a variable to pass to unlockfile().
|
|
||||||
"""
|
|
||||||
config.logger.debug("take lockfile %s", name)
|
|
||||||
dirname = os.path.dirname(name)
|
|
||||||
mkdirhier(dirname)
|
|
||||||
|
|
||||||
if not os.access(dirname, os.W_OK):
|
|
||||||
logger.error("Unable to acquire lock '%s', directory is not writable",
|
|
||||||
name)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
operation = fcntl.LOCK_EX
|
|
||||||
if shared:
|
|
||||||
operation = fcntl.LOCK_SH
|
|
||||||
if not retry:
|
|
||||||
operation = operation | fcntl.LOCK_NB
|
|
||||||
|
|
||||||
while True:
|
|
||||||
# If we leave the lockfiles lying around there is no problem
|
|
||||||
# but we should clean up after ourselves. This gives potential
|
|
||||||
# for races though. To work around this, when we acquire the lock
|
|
||||||
# we check the file we locked was still the lock file on disk.
|
|
||||||
# by comparing inode numbers. If they don't match or the lockfile
|
|
||||||
# no longer exists, we start again.
|
|
||||||
|
|
||||||
# This implementation is unfair since the last person to request the
|
|
||||||
# lock is the most likely to win it.
|
|
||||||
|
|
||||||
# pylint: disable=broad-except
|
|
||||||
# we disable the broad-except because we want to actually catch all possible exceptions
|
|
||||||
try:
|
|
||||||
lock_file = open(name, 'a+')
|
|
||||||
fileno = lock_file.fileno()
|
|
||||||
fcntl.flock(fileno, operation)
|
|
||||||
statinfo = os.fstat(fileno)
|
|
||||||
if os.path.exists(lock_file.name):
|
|
||||||
statinfo2 = os.stat(lock_file.name)
|
|
||||||
if statinfo.st_ino == statinfo2.st_ino:
|
|
||||||
return lock_file
|
|
||||||
lock_file.close()
|
|
||||||
except Exception as exc:
|
|
||||||
try:
|
|
||||||
lock_file.close()
|
|
||||||
except Exception as exc2:
|
|
||||||
config.logger.error("Failed to close the lockfile: %s", exc2)
|
|
||||||
config.logger.error("Failed to acquire the lockfile: %s", exc)
|
|
||||||
if not retry:
|
|
||||||
return None
|
|
||||||
|
|
||||||
def unlockfile(lock_file):
|
|
||||||
"""
|
|
||||||
Unlock a file locked using lockfile()
|
|
||||||
"""
|
|
||||||
try:
|
|
||||||
# If we had a shared lock, we need to promote to exclusive before
|
|
||||||
# removing the lockfile. Attempt this, ignore failures.
|
|
||||||
fcntl.flock(lock_file.fileno(), fcntl.LOCK_EX|fcntl.LOCK_NB)
|
|
||||||
os.unlink(lock_file.name)
|
|
||||||
except (IOError, OSError):
|
|
||||||
pass
|
|
||||||
fcntl.flock(lock_file.fileno(), fcntl.LOCK_UN)
|
|
||||||
lock_file.close()
|
|
||||||
|
|
||||||
#ENDOFCOPY
|
|
||||||
|
|
||||||
|
|
||||||
def mk_lock_filename():
|
|
||||||
our_name = os.path.basename(__file__)
|
|
||||||
our_name = ".%s" % ".".join(reversed(our_name.split(".")))
|
|
||||||
return config.LOCKFILE + our_name
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
class ShellCmdException(Exception):
|
|
||||||
pass
|
|
||||||
|
|
||||||
def run_shell_cmd(command, cwd=None):
|
|
||||||
if cwd is None:
|
|
||||||
cwd = os.getcwd()
|
|
||||||
|
|
||||||
config.logger.debug("_shellcmd: (%s) %s", cwd, command)
|
|
||||||
process = subprocess.Popen(command, cwd=cwd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
|
||||||
(out, err) = process.communicate()
|
|
||||||
process.wait()
|
|
||||||
if process.returncode:
|
|
||||||
if len(err) == 0:
|
|
||||||
err = "command: %s \n%s" % (command, out)
|
|
||||||
else:
|
|
||||||
err = "command: %s \n%s" % (command, err)
|
|
||||||
config.logger.warning("_shellcmd: error \n%s\n%s", out, err)
|
|
||||||
raise ShellCmdException(err)
|
|
||||||
else:
|
|
||||||
#config.logger.debug("localhostbecontroller: shellcmd success\n%s" % out)
|
|
||||||
return out
|
|
||||||
|
|
|
@ -1,115 +0,0 @@
|
||||||
#!/usr/bin/python
|
|
||||||
|
|
||||||
# ex:ts=4:sw=4:sts=4:et
|
|
||||||
# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
|
|
||||||
#
|
|
||||||
# Copyright (C) 2015 Alexandru Damian for Intel Corp.
|
|
||||||
#
|
|
||||||
# This program is free software; you can redistribute it and/or modify
|
|
||||||
# it under the terms of the GNU General Public License version 2 as
|
|
||||||
# published by the Free Software Foundation.
|
|
||||||
#
|
|
||||||
# This program is distributed in the hope that it will be useful,
|
|
||||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
||||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
||||||
# GNU General Public License for more details.
|
|
||||||
#
|
|
||||||
# You should have received a copy of the GNU General Public License along
|
|
||||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
|
||||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
|
||||||
|
|
||||||
|
|
||||||
# Test definitions. The runner will look for and auto-discover the tests
|
|
||||||
# no matter what they file are they in, as long as they are in the same directory
|
|
||||||
# as this file.
|
|
||||||
|
|
||||||
import unittest
|
|
||||||
from shellutils import run_shell_cmd, ShellCmdException
|
|
||||||
import config
|
|
||||||
|
|
||||||
import pexpect
|
|
||||||
import sys, os, signal, time
|
|
||||||
|
|
||||||
class Test00PyCompilable(unittest.TestCase):
|
|
||||||
''' Verifies that all Python files are syntactically correct '''
|
|
||||||
def test_compile_file(self):
|
|
||||||
try:
|
|
||||||
run_shell_cmd("find . -name *py -type f -print0 | xargs -0 -n1 -P20 python -m py_compile", config.TESTDIR)
|
|
||||||
except ShellCmdException as exc:
|
|
||||||
self.fail("Error compiling python files: %s" % (exc))
|
|
||||||
|
|
||||||
def test_pylint_file(self):
|
|
||||||
try:
|
|
||||||
run_shell_cmd(r"find . -iname \"*\.py\" -type f -print0 | PYTHONPATH=${PYTHONPATH}:. xargs -r -0 -n1 pylint --load-plugins pylint_django -E --reports=n 2>&1", cwd=config.TESTDIR + "/bitbake/lib/toaster")
|
|
||||||
except ShellCmdException as exc:
|
|
||||||
self.fail("Pylint fails: %s\n" % exc)
|
|
||||||
|
|
||||||
class Test01PySystemStart(unittest.TestCase):
|
|
||||||
''' Attempts to start Toaster, verify that it is succesfull, and stop it '''
|
|
||||||
def setUp(self):
|
|
||||||
run_shell_cmd("bash -c 'rm -f build/*log'")
|
|
||||||
|
|
||||||
def test_start_interactive_mode(self):
|
|
||||||
try:
|
|
||||||
run_shell_cmd("bash -c 'source %s/oe-init-build-env && source toaster start webport=%d && source toaster stop'" % (config.TESTDIR, config.TOASTER_PORT), config.TESTDIR)
|
|
||||||
except ShellCmdException as exc:
|
|
||||||
self.fail("Failed starting interactive mode: %s" % (exc))
|
|
||||||
|
|
||||||
def test_start_managed_mode(self):
|
|
||||||
try:
|
|
||||||
run_shell_cmd("%s/bitbake/bin/toaster webport=%d nobrowser & sleep 10 && curl http://localhost:%d/ && kill -2 %%1" % (config.TESTDIR, config.TOASTER_PORT, config.TOASTER_PORT), config.TESTDIR)
|
|
||||||
except ShellCmdException as exc:
|
|
||||||
self.fail("Failed starting managed mode: %s" % (exc))
|
|
||||||
|
|
||||||
class Test02HTML5Compliance(unittest.TestCase):
|
|
||||||
def setUp(self):
|
|
||||||
self.origdir = os.getcwd()
|
|
||||||
self.crtdir = os.path.dirname(config.TESTDIR)
|
|
||||||
self.cleanup_database = False
|
|
||||||
os.chdir(self.crtdir)
|
|
||||||
if not os.path.exists(os.path.join(self.crtdir, "toaster.sqlite")):
|
|
||||||
self.cleanup_database = True
|
|
||||||
run_shell_cmd("%s/bitbake/lib/toaster/manage.py syncdb --noinput" % config.TESTDIR)
|
|
||||||
run_shell_cmd("%s/bitbake/lib/toaster/manage.py migrate orm" % config.TESTDIR)
|
|
||||||
run_shell_cmd("%s/bitbake/lib/toaster/manage.py migrate bldcontrol" % config.TESTDIR)
|
|
||||||
run_shell_cmd("%s/bitbake/lib/toaster/manage.py loadconf %s/meta-yocto/conf/toasterconf.json" % (config.TESTDIR, config.TESTDIR))
|
|
||||||
run_shell_cmd("%s/bitbake/lib/toaster/manage.py lsupdates" % config.TESTDIR)
|
|
||||||
|
|
||||||
setup = pexpect.spawn("%s/bitbake/lib/toaster/manage.py checksettings" % config.TESTDIR)
|
|
||||||
setup.logfile = sys.stdout
|
|
||||||
setup.expect(r".*or type the full path to a different directory: ")
|
|
||||||
setup.sendline('')
|
|
||||||
setup.sendline('')
|
|
||||||
setup.expect(r".*or type the full path to a different directory: ")
|
|
||||||
setup.sendline('')
|
|
||||||
setup.expect(r"Enter your option: ")
|
|
||||||
setup.sendline('0')
|
|
||||||
|
|
||||||
self.child = pexpect.spawn("bash", ["%s/bitbake/bin/toaster" % config.TESTDIR, "webport=%d" % config.TOASTER_PORT, "nobrowser"], cwd=self.crtdir)
|
|
||||||
self.child.logfile = sys.stdout
|
|
||||||
self.child.expect("Toaster is now running. You can stop it with Ctrl-C")
|
|
||||||
|
|
||||||
def test_html5_compliance(self):
|
|
||||||
import urllist, urlcheck
|
|
||||||
results = {}
|
|
||||||
for url in urllist.URLS:
|
|
||||||
results[url] = urlcheck.validate_html5(config.TOASTER_BASEURL + url)
|
|
||||||
|
|
||||||
failed = []
|
|
||||||
for url in results:
|
|
||||||
if results[url][1] != 0:
|
|
||||||
failed.append((url, results[url]))
|
|
||||||
|
|
||||||
|
|
||||||
self.assertTrue(len(failed) == 0, "Not all URLs validate: \n%s " % "\n".join(["".join(str(x)) for x in failed]))
|
|
||||||
|
|
||||||
#(config.TOASTER_BASEURL + url, status, errors, warnings))
|
|
||||||
|
|
||||||
def tearDown(self):
|
|
||||||
while self.child.isalive():
|
|
||||||
self.child.kill(signal.SIGINT)
|
|
||||||
time.sleep(1)
|
|
||||||
os.chdir(self.origdir)
|
|
||||||
toaster_sqlite_path = os.path.join(self.crtdir, "toaster.sqlite")
|
|
||||||
if self.cleanup_database and os.path.exists(toaster_sqlite_path):
|
|
||||||
os.remove(toaster_sqlite_path)
|
|
|
@ -1,155 +0,0 @@
|
||||||
#!/usr/bin/python
|
|
||||||
|
|
||||||
# Copyright
|
|
||||||
|
|
||||||
# DESCRIPTION
|
|
||||||
# This is script for running all selected toaster cases on
|
|
||||||
# selected web browsers manifested in toaster_test.cfg.
|
|
||||||
|
|
||||||
# 1. How to start toaster in yocto:
|
|
||||||
# $ source poky/oe-init-build-env
|
|
||||||
# $ source toaster start
|
|
||||||
# $ bitbake core-image-minimal
|
|
||||||
|
|
||||||
# 2. How to install selenium on Ubuntu:
|
|
||||||
# $ sudo apt-get install scrot python-pip
|
|
||||||
# $ sudo pip install selenium
|
|
||||||
|
|
||||||
# 3. How to install selenium addon in firefox:
|
|
||||||
# Download the lastest firefox addon from http://release.seleniumhq.org/selenium-ide/
|
|
||||||
# Then install it. You can also install firebug and firepath addon
|
|
||||||
|
|
||||||
# 4. How to start writing a new case:
|
|
||||||
# All you need to do is to implement the function test_xxx() and pile it on.
|
|
||||||
|
|
||||||
# 5. How to test with Chrome browser
|
|
||||||
# Download/install chrome on host
|
|
||||||
# Download chromedriver from https://code.google.com/p/chromedriver/downloads/list according to your host type
|
|
||||||
# put chromedriver in PATH, (e.g. /usr/bin/, bear in mind to chmod)
|
|
||||||
# For windows host, you may put chromedriver.exe in the same directory as chrome.exe
|
|
||||||
|
|
||||||
import unittest, sys, os, platform
|
|
||||||
import ConfigParser
|
|
||||||
import argparse
|
|
||||||
from toaster_automation_test import toaster_cases
|
|
||||||
|
|
||||||
|
|
||||||
def get_args_parser():
|
|
||||||
description = "Script that runs toaster auto tests."
|
|
||||||
parser = argparse.ArgumentParser(description=description)
|
|
||||||
parser.add_argument('--run-all-tests', required=False, action="store_true", dest="run_all_tests", default=False,
|
|
||||||
help='Run all tests.')
|
|
||||||
parser.add_argument('--run-suite', required=False, dest='run_suite', default=False,
|
|
||||||
help='run suite (defined in cfg file)')
|
|
||||||
|
|
||||||
return parser
|
|
||||||
|
|
||||||
|
|
||||||
def get_tests():
|
|
||||||
testslist = []
|
|
||||||
|
|
||||||
prefix = 'toaster_automation_test.toaster_cases'
|
|
||||||
|
|
||||||
for t in dir(toaster_cases):
|
|
||||||
if t.startswith('test_'):
|
|
||||||
testslist.append('.'.join((prefix, t)))
|
|
||||||
|
|
||||||
return testslist
|
|
||||||
|
|
||||||
|
|
||||||
def get_tests_from_cfg(suite=None):
|
|
||||||
|
|
||||||
testslist = []
|
|
||||||
config = ConfigParser.SafeConfigParser()
|
|
||||||
config.read('toaster_test.cfg')
|
|
||||||
|
|
||||||
if suite is not None:
|
|
||||||
target_suite = suite.lower()
|
|
||||||
|
|
||||||
# TODO: if suite is valid suite
|
|
||||||
|
|
||||||
else:
|
|
||||||
target_suite = platform.system().lower()
|
|
||||||
|
|
||||||
try:
|
|
||||||
tests_from_cfg = eval(config.get('toaster_test_' + target_suite, 'test_cases'))
|
|
||||||
except:
|
|
||||||
print('Failed to get test cases from cfg file. Make sure the format is correct.')
|
|
||||||
return None
|
|
||||||
|
|
||||||
prefix = 'toaster_automation_test.toaster_cases.test_'
|
|
||||||
for t in tests_from_cfg:
|
|
||||||
testslist.append(prefix + str(t))
|
|
||||||
|
|
||||||
return testslist
|
|
||||||
|
|
||||||
def main():
|
|
||||||
|
|
||||||
# In case this script is called from other directory
|
|
||||||
os.chdir(os.path.abspath(sys.path[0]))
|
|
||||||
|
|
||||||
parser = get_args_parser()
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
if args.run_all_tests:
|
|
||||||
testslist = get_tests()
|
|
||||||
elif args.run_suite:
|
|
||||||
testslist = get_tests_from_cfg(args.run_suite)
|
|
||||||
os.environ['TOASTER_SUITE'] = args.run_suite
|
|
||||||
else:
|
|
||||||
testslist = get_tests_from_cfg()
|
|
||||||
|
|
||||||
if not testslist:
|
|
||||||
print('Failed to get test cases.')
|
|
||||||
exit(1)
|
|
||||||
|
|
||||||
suite = unittest.TestSuite()
|
|
||||||
loader = unittest.TestLoader()
|
|
||||||
loader.sortTestMethodsUsing = None
|
|
||||||
runner = unittest.TextTestRunner(verbosity=2, resultclass=buildResultClass(args))
|
|
||||||
|
|
||||||
for test in testslist:
|
|
||||||
try:
|
|
||||||
suite.addTests(loader.loadTestsFromName(test))
|
|
||||||
except:
|
|
||||||
return 1
|
|
||||||
|
|
||||||
result = runner.run(suite)
|
|
||||||
|
|
||||||
if result.wasSuccessful():
|
|
||||||
return 0
|
|
||||||
else:
|
|
||||||
return 1
|
|
||||||
|
|
||||||
|
|
||||||
def buildResultClass(args):
|
|
||||||
"""Build a Result Class to use in the testcase execution"""
|
|
||||||
|
|
||||||
class StampedResult(unittest.TextTestResult):
|
|
||||||
"""
|
|
||||||
Custom TestResult that prints the time when a test starts. As toaster-auto
|
|
||||||
can take a long time (ie a few hours) to run, timestamps help us understand
|
|
||||||
what tests are taking a long time to execute.
|
|
||||||
"""
|
|
||||||
def startTest(self, test):
|
|
||||||
import time
|
|
||||||
self.stream.write(time.strftime("%Y-%m-%d %H:%M:%S", time.localtime()) + " - ")
|
|
||||||
super(StampedResult, self).startTest(test)
|
|
||||||
|
|
||||||
return StampedResult
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
|
|
||||||
try:
|
|
||||||
ret = main()
|
|
||||||
except:
|
|
||||||
ret = 1
|
|
||||||
import traceback
|
|
||||||
traceback.print_exc()
|
|
||||||
finally:
|
|
||||||
if os.getenv('TOASTER_SUITE'):
|
|
||||||
del os.environ['TOASTER_SUITE']
|
|
||||||
sys.exit(ret)
|
|
||||||
|
|
||||||
|
|
File diff suppressed because it is too large
Load Diff
|
@ -1,25 +0,0 @@
|
||||||
# Configuration file for toaster_test
|
|
||||||
# Sorted by different host type
|
|
||||||
|
|
||||||
# test browser could be: firefox; chrome; ie(still under development)
|
|
||||||
# logging_level could be: CRITICAL; ERROR; WARNING; INFO; DEBUG; NOTSET
|
|
||||||
|
|
||||||
|
|
||||||
[toaster_test_linux]
|
|
||||||
toaster_url = 'http://127.0.0.1:8000'
|
|
||||||
test_browser = 'firefox'
|
|
||||||
test_cases = [946]
|
|
||||||
logging_level = 'INFO'
|
|
||||||
|
|
||||||
|
|
||||||
[toaster_test_windows]
|
|
||||||
toaster_url = 'http://127.0.0.1:8000'
|
|
||||||
test_browser = ['ie', 'firefox', 'chrome']
|
|
||||||
test_cases = [901, 902, 903]
|
|
||||||
logging_level = 'DEBUG'
|
|
||||||
|
|
||||||
[toaster_test_darwin]
|
|
||||||
toaster_url = 'http://127.0.0.1:8000'
|
|
||||||
test_browser = 'firefox'
|
|
||||||
test_cases = [901, 902, 903, 904, 906, 910, 911, 912, 913, 914, 915, 916, 923, 924, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 955, 956]
|
|
||||||
logging_level = 'INFO'
|
|
|
@ -1,53 +0,0 @@
|
||||||
from __future__ import print_function
|
|
||||||
import sys
|
|
||||||
|
|
||||||
import httplib2
|
|
||||||
import config
|
|
||||||
import urllist
|
|
||||||
|
|
||||||
|
|
||||||
config.logger.info("Testing %s with %s", config.TOASTER_BASEURL, config.W3C_VALIDATOR)
|
|
||||||
|
|
||||||
def validate_html5(url):
|
|
||||||
http_client = httplib2.Http(None)
|
|
||||||
status = "Failed"
|
|
||||||
errors = -1
|
|
||||||
warnings = -1
|
|
||||||
|
|
||||||
urlrequest = config.W3C_VALIDATOR+url
|
|
||||||
|
|
||||||
# pylint: disable=broad-except
|
|
||||||
# we disable the broad-except because we want to actually catch all possible exceptions
|
|
||||||
try:
|
|
||||||
resp, _ = http_client.request(urlrequest, "HEAD")
|
|
||||||
if resp['x-w3c-validator-status'] != "Abort":
|
|
||||||
status = resp['x-w3c-validator-status']
|
|
||||||
errors = int(resp['x-w3c-validator-errors'])
|
|
||||||
warnings = int(resp['x-w3c-validator-warnings'])
|
|
||||||
|
|
||||||
if status == 'Invalid':
|
|
||||||
config.logger.warning("Failed %s is %s\terrors %s warnings %s (check at %s)", url, status, errors, warnings, urlrequest)
|
|
||||||
else:
|
|
||||||
config.logger.debug("OK! %s", url)
|
|
||||||
|
|
||||||
except Exception as exc:
|
|
||||||
config.logger.warning("Failed validation call: %s", exc)
|
|
||||||
return (status, errors, warnings)
|
|
||||||
|
|
||||||
|
|
||||||
def print_validation(url):
|
|
||||||
status, errors, warnings = validate_html5(url)
|
|
||||||
config.logger.error("url %s is %s\terrors %s warnings %s (check at %s)", url, status, errors, warnings, config.W3C_VALIDATOR+url)
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
print("Testing %s with %s" % (config.TOASTER_BASEURL, config.W3C_VALIDATOR))
|
|
||||||
|
|
||||||
if len(sys.argv) > 1:
|
|
||||||
print_validation(sys.argv[1])
|
|
||||||
else:
|
|
||||||
for url in urllist.URLS:
|
|
||||||
print_validation(config.TOASTER_BASEURL+url)
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
|
@ -1,39 +0,0 @@
|
||||||
URLS = [
|
|
||||||
'toastergui/landing/',
|
|
||||||
'toastergui/builds/',
|
|
||||||
'toastergui/build/1',
|
|
||||||
'toastergui/build/1/tasks/',
|
|
||||||
'toastergui/build/1/tasks/1/',
|
|
||||||
'toastergui/build/1/task/1',
|
|
||||||
'toastergui/build/1/recipes/',
|
|
||||||
'toastergui/build/1/recipe/1/active_tab/1',
|
|
||||||
'toastergui/build/1/recipe/1',
|
|
||||||
'toastergui/build/1/recipe_packages/1',
|
|
||||||
'toastergui/build/1/packages/',
|
|
||||||
'toastergui/build/1/package/1',
|
|
||||||
'toastergui/build/1/package_built_dependencies/1',
|
|
||||||
'toastergui/build/1/package_included_detail/1/1',
|
|
||||||
'toastergui/build/1/package_included_dependencies/1/1',
|
|
||||||
'toastergui/build/1/package_included_reverse_dependencies/1/1',
|
|
||||||
'toastergui/build/1/target/1',
|
|
||||||
'toastergui/build/1/target/1/targetpkg',
|
|
||||||
'toastergui/build/1/target/1/dirinfo',
|
|
||||||
'toastergui/build/1/target/1/dirinfo_filepath/_/bin/bash',
|
|
||||||
'toastergui/build/1/configuration',
|
|
||||||
'toastergui/build/1/configvars',
|
|
||||||
'toastergui/build/1/buildtime',
|
|
||||||
'toastergui/build/1/cpuusage',
|
|
||||||
'toastergui/build/1/diskio',
|
|
||||||
'toastergui/build/1/target/1/packagefile/1',
|
|
||||||
'toastergui/newproject/',
|
|
||||||
'toastergui/projects/',
|
|
||||||
'toastergui/project/1',
|
|
||||||
'toastergui/project/1/configuration',
|
|
||||||
'toastergui/project/1/builds/',
|
|
||||||
'toastergui/project/1/layers/',
|
|
||||||
'toastergui/project/1/layer/1',
|
|
||||||
'toastergui/project/1/importlayer',
|
|
||||||
'toastergui/project/1/targets/',
|
|
||||||
'toastergui/project/1/machines/',
|
|
||||||
'toastergui/',
|
|
||||||
]
|
|
Loading…
Reference in New Issue