- This allows those tests to be invoked on console and not just in CI pipelines.
Tests:
- Running make check works.
- CI pipeline succeeds.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
[jvalleroy: Remove phony test target]
Signed-off-by: James Valleroy <jvalleroy@mailbox.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Tests:
- All tests in patch series have been done with this patch applied
- Unit tests pass
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Tests:
- Fail a functional test using assert False. Notice that the screenshot is taken
in ./screenshots directory.
- Pass --splinter-screenshot-dir=foo and notice that the screenshot is taken in
./foo directory.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
pytest_splinter has an issue with creating screenshots when using
session_browser instead of a browser fixture. Implement a workaround.
See: https://github.com/pytest-dev/pytest-splinter/pull/157
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Create the 'heavy' mark for tests that take long to run.
- Skip the tests marked as 'heavy' by default.
- Enable functional and heavy tests by setting the environment variable
EXTENDED_TESTING=1.
- Update HACKING.md accordingly.
Signed-off-by: Fioddor Superconcentrado <fioddor@gmail.com>
[sunil: Minor indentation]
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
- Derived from the often repeated fixture code in various test_actions.py files.
- Don't use the deprecated imp module. Instead use importlib. imp module will be
removed in future versions of Python.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Veiko Aasa <veiko17@disroot.org>
- Use the test configuration only when needed. This simplifies having to load
test configuration properly for action tests.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- The configuration module defaults to values in the production configuration
file.
- If the file is found, it is read and the read values overwrite the defaults.
If the file is not found, no error is raised. This allows us to not ship the
configuration file. User may create the configuration if they want to change the
defaults. This eases upgrades when configuration is edited. This also make
FreedomBox robust to deployments where /etc/ is not populated by default such as
OSTree. It is also a good practice for daemons as followed by the likes of
systemd.
- If the file partly populated only the values read override the defaults and
the remaining values don't change. This allows the user to write simpler
configuration file.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Splinter/selenium have implicit and explicit waiting time. Implicit wait time
will make every negative lookup wait for about 3 seconds before it actually
fails. Because we ensure missing elements in quite a few places, this introduces
many 3 seconds wait periods during testing. Remove it instead rely on explicit
waiting whenever needed.
- Explicit wait time is only used during explicitly requests waiting conditions.
In a loop the API waits for a maximum of timeout period until a given condition
is satisfied. Each time the condition is checked, it goes into sleep for
explicit wait period amount of time. This is typically a second or so. Since we
are impatient, make it 0.1 instead.
- Also make sure that whenever a page is visit()ed, we automatically wait until
the page is fully loaded by overriding the splinter wait condition. Otherwise,
we will need to introduce waiting code in a lot of places.
- Using document.readyState == complete is a better check to ensure that a page
is fully loaded. If we proceed with the page 'loading' or 'interactive' state,
we will have to change a lot of code to make it wait.
- Handle Apache restarts when waiting for page load. The error page apparently
is never reaches document.readyState == 'complete'. So, if an error page is
encountered, always reload.
- While waiting for installation, ensure that we atomically check that page has
loaded fully and the installation progress is not being shown. Otherwise, there
would be race condition due to installation page refreshing itself.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Move non-reusable app specific step definitions and helper methods into
<app>/tests/test_functional.py.
- Merge reusable helper methods into plinth.tests.functional
- Merge reusable step definitions into plinth.tests.functional.step_definitions
- avahi, datetime, ikiwiki: Reuse common methods to avoid repetition. Avoid
mapping from app nicknames to actual app names.
- deluge, transmission: Make a copy of sample.torrent for each app to avoid
clogging common place.
- Implement functional.visit() to simplify a lot of browser.visit() calls.
- Ensure that name of the mark on functional tests for an app is same as name of
the app. This will help with predicting the mark when running tests for a
particular app.
Tests performed:
- Run all functional tests.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Joseph Nuthalapati <njoseph@riseup.net>
- Add pytest hooks to ignore all functional tests if pytest_bdd is not
installed.
- Update pytest hooks to skip tests in file named 'test_functional.py' if
--include-functional argument is not provided.
- Move functional_tests/install.py into plinth/tests/functional and update
reference in Vagrantfile.
- Move scenario files into individual app folders. Rename them after the app
they are testing. Merge TODO items listed in todo.org into corresponding feature
files.
- Add test_functional.py in each app to build tests from the features file using
pytest_bdd.
- Move all step_definitions, support and data into plinth/tests/functional/.
Include all step_definitions from conftest.py. Update to relative imports
instead of absolute imports.
Tests performed:
- Run py.test-3 --collect-only shows all functional tests and lists 574 tests.
No errors show that name of feature files are correct. The number says that all
functional test features are included.
- Remove pytest_bdd (or modify the import name) and run py.test-3 --collect-only
skips collecting all functional tests and shows only 300+ tests.
- Run functional tests for a few apps with py.test-3 --include-functional -m
app. For storage, deluge.
- Run unit tests with py.test-3. Functional tests are listed by skipped.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Joseph Nuthalapati <njoseph@riseup.net>
Fixtures cannot be currently included into other fixtures by using
@pytest.mark.fixtures('fixture_name')
They have to be included as parameters instead.
See bug: https://github.com/pytest-dev/pytest/issues/3664
Also increase the scope of needs_root to the highest, i.e. session, so that it
can be used by any kind of fixture.
Signed-off-by: Joseph Nuthalapati <njoseph@thoughtworks.com>
- Create option --include-functional to run functional tests. Otherwise, they
are disabled by default. If pytest-bdd is not installed, functional tests are
not discovered at all.
- Make pytest-django discover the setting files by creating dummy manage.py in
top level directory.
- Make pytest run as './setup.py pytest'. Add alias from './setup.py test'. This
requires pytest-runner package.
- Merge .gitignore files from functional_tests/
- Update gitlab-ci.yml to run tests with coverage using pytest.
- Update HACKING.md to suggest using py.test-3 instead of old way of running.
Merge functional tests README.md into HACKING.md.
- Remove execution wrapper runtests.py as pytest-django is able to configure
Django settings before execution of tests. Update tests to explicitly ask for
Django database as database access is denied by default.
- Replace usage of python3-coverage with python3-pytest-coverage. Execution
wrappers are not required.
- Add build dependencies on pytest modules.
- Let all warnings be shown after running tests.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
