53 Commits

Author SHA1 Message Date
Sunil Mohan Adapa
3255a7e658
backups: schedule: tests: Fix failures due to long test run
Closes: #2023.

When importing the test module, datetime.now() is executed and value is kept. If
a test suite runs for a long time, the time deltas are being calculated much
later when the test case runs. This creates an difference in expected different
between the two values.

Fix this by completely removing all uses of time relative to current date time.
Only use absolute date time values. This should not reduce the effectiveness of
the test cases.

Tests performed:

- Rerun unit tests for backups module.

Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
2021-01-30 14:29:05 -05:00
Sunil Mohan Adapa
eb526275c7
backups: Add UI to edit schedules
Closes: #1529.

Tests performed:

- Functional tests for backups app work.

- Functional tests for backup of several apps work.

Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
2021-01-21 20:55:27 -05:00
Sunil Mohan Adapa
dfb8b69531
backups: Introduce backup scheduling
Tests performed:

- Unit tests work.

Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
2021-01-21 20:49:15 -05:00
Sunil Mohan Adapa
01e00cdde4
backups: Allow comments to be added to archives during backup
Tests performed:

- Schedules are able to store and retrieve comments properly. Information about
schedule backups stored in comments is extracted properly.

- Unit tests run.

Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
2021-01-21 20:33:47 -05:00
Sunil Mohan Adapa
d3f8e815c1
backups: tests: Fix a typo in test case name
Tests performed:

- Ran all backup related unit tests.

Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
2021-01-21 20:33:41 -05:00
Sunil Mohan Adapa
fb1898befc
backups: Use the backup component in all apps
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Veiko Aasa <veiko17@disroot.org>
2021-01-04 13:47:38 +02:00
Sunil Mohan Adapa
f630fb6059
backups: Add new component for backup and restore
- Component performs the validation instead of each app calling the validator.

- Retain the schema for declaring backup information.

- Suitable for usage directly in the backup/restore operation packets instead of
using BackupApp class.

- Backup/restore hooks can be handled by overriding this component instead of
module level methods.

Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Veiko Aasa <veiko17@disroot.org>
2021-01-04 13:47:37 +02:00
Benjamin Ortiz
5e6a661648
backups: Allow remote repository usernames to start with numbers
Although useradd recommends starting with either a lowercase letter or an
underscore, there is nothing that consistently adheres to this recommendation
across systems. Because some systems do not follow this recommendation and this
recommendation is not a hard requirement, we should not prevent connections at
the validation stage.

Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
2020-07-12 19:26:54 -07:00
Sunil Mohan Adapa
80dff7bf9c
tests: functional: Re-organize step definitions and helper methods
- Move non-reusable app specific step definitions and helper methods into
<app>/tests/test_functional.py.

- Merge reusable helper methods into plinth.tests.functional

- Merge reusable step definitions into plinth.tests.functional.step_definitions

- avahi, datetime, ikiwiki: Reuse common methods to avoid repetition. Avoid
mapping from app nicknames to actual app names.

- deluge, transmission: Make a copy of sample.torrent for each app to avoid
clogging common place.

- Implement functional.visit() to simplify a lot of browser.visit() calls.

- Ensure that name of the mark on functional tests for an app is same as name of
the app. This will help with predicting the mark when running tests for a
particular app.

Tests performed:

- Run all functional tests.

Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Joseph Nuthalapati <njoseph@riseup.net>
2020-05-22 22:52:40 +05:30
Sunil Mohan Adapa
80d67c2054
tests: functional: Merge into main source hierarchy
- Add pytest hooks to ignore all functional tests if pytest_bdd is not
installed.

- Update pytest hooks to skip tests in file named 'test_functional.py' if
--include-functional argument is not provided.

- Move functional_tests/install.py into plinth/tests/functional and update
reference in Vagrantfile.

- Move scenario files into individual app folders. Rename them after the app
they are testing. Merge TODO items listed in todo.org into corresponding feature
files.

- Add test_functional.py in each app to build tests from the features file using
pytest_bdd.

- Move all step_definitions, support and data into plinth/tests/functional/.
Include all step_definitions from conftest.py. Update to relative imports
instead of absolute imports.

Tests performed:

- Run py.test-3 --collect-only shows all functional tests and lists 574 tests.
No errors show that name of feature files are correct. The number says that all
functional test features are included.

- Remove pytest_bdd (or modify the import name) and run py.test-3 --collect-only
skips collecting all functional tests and shows only 300+ tests.

- Run functional tests for a few apps with py.test-3 --include-functional -m
app. For storage, deluge.

- Run unit tests with py.test-3. Functional tests are listed by skipped.

Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Joseph Nuthalapati <njoseph@riseup.net>
2020-05-22 21:26:05 +05:30
Sunil Mohan Adapa
9368504da5
*.py: Use SPDX license identifier
Reviewed-by: Veiko Aasa <veiko17@disroot.org>
2020-02-19 14:38:55 +02:00
Alice Kile
eb83e00011
fix formatting issues
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
2019-12-07 13:08:35 -05:00
Sunil Mohan Adapa
7467fa4553
backups: Separate repository loading from instantiation
The constructor of the Repository object is being used for two distinct
purposes. One is to load the object from database and other to instantiate it
with parameter such that it can be saved to database. Separating the two usages
to different methods simplifies code and parameter passing for consumers.

Also turn some class specific constants from globals to class constants.

Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
2019-09-07 12:52:49 -04:00
Sunil Mohan Adapa
5b01689a92
backups: Clarify two separate uses of name create_repository
It is being used to mean initialize the borg repository as well as creating an
instance of the repository class object. Use 'initialize' for former and
'get_instance' for latter.

Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
2019-09-07 12:52:46 -04:00
Sunil Mohan Adapa
d05bbab751
backups: Rename repo_path to borg_path for clarity
borg_path clearly signifies that it is to be used by borg and consumers of the
class will not be confused by it.

Also rename some repo_path variables in test cases.

Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
2019-09-07 12:52:36 -04:00
Sunil Mohan Adapa
0df07f5cf2
backups: Rename network_storage module to store
Same module is being used to store/retrieve local disk repositories also.

Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
2019-09-07 12:52:17 -04:00
Joseph Nuthalpati
8a93b5b90c
backups: Allow adding backup repositories on multiple disks
Signed-off-by: Joseph Nuthalapati <njoseph@riseup.net>
[sunil@medhas.org Fix showing form choices, undo blank line removals]
[sunil@medhas.org Fix typo in tooltip for add repo button]
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
2019-09-07 12:51:40 -04:00
Nikolas Nyby
ace339eabf
Introduce flake8 checking - #58
This introduces flake8 and fixes a bunch of flake8 errors.

flake8 is run with: ./venv/bin/flake8 plinth
if you're using a python3 venv.

We can eventually further integrate this with gitlab ci.

https://salsa.debian.org/freedombox-team/plinth/issues/58

Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
2019-07-24 14:01:54 -07:00
Sunil Mohan Adapa
dd5d93637f
backups: Don't send passphrase on the command line
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Joseph Nuthalapati <njoseph@thoughtworks.com>
2019-07-03 12:44:54 +05:30
Sunil Mohan Adapa
9c8674baa3
backup: Allow SSH directory paths with : in them
- From SSH manual, a path may contain : if it is presented as absolute path.
  Update regular expression for splitting accordingly.

- Allow Null paths similar to SSH.

- Perform a full regular expression match when splitting path.

- Simplify regular expression.

- Update tests.

Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Joseph Nuthalapati <njoseph@thoughtworks.com>
2019-07-03 12:43:28 +05:30
Joseph Nuthalapati
bd1874d774
backups: Add regex validation for ssh_repository field
Signed-off-by: Joseph Nuthalapati <njoseph@thoughtworks.com>
2019-07-03 12:43:08 +05:30
Joseph Nuthalapati
3a6dcbe7a7
Verify SSH hostkey before mounting
Signed-off-by: Joseph Nuthalapati <njoseph@thoughtworks.com>
2019-07-03 12:42:41 +05:30
Joseph Nuthalapati
0b43caf81d
Add SSH hostkey verification
Signed-off-by: Joseph Nuthalapati <njoseph@thoughtworks.com>
2019-07-03 12:42:25 +05:30
Joseph Nuthalapati
f85e7829b9
backups: tests: Fix issue with usage of fixture 'needs_root'
Fixtures cannot be currently included into other fixtures by using
@pytest.mark.fixtures('fixture_name')
They have to be included as parameters instead.
See bug: https://github.com/pytest-dev/pytest/issues/3664

Also increase the scope of needs_root to the highest, i.e. session, so that it
can be used by any kind of fixture.

Signed-off-by: Joseph Nuthalapati <njoseph@thoughtworks.com>
2019-07-03 12:42:20 +05:30
Joseph Nuthalapati
f4f9e2c465
backups: Skipping tests temporarily
Signed-off-by: Joseph Nuthalapati <njoseph@thoughtworks.com>
2019-07-03 12:42:14 +05:30
Joseph Nuthalapati
ee21dcfbb8
backups: Avoid using sudo in tests
Signed-off-by: Joseph Nuthalapati <njoseph@thoughtworks.com>
2019-07-03 12:42:10 +05:30
Joseph Nuthalapati
d451b54286
backups: Add test for adding ssh remotes
Signed-off-by: Joseph Nuthalapati <njoseph@thoughtworks.com>
2019-07-03 12:42:02 +05:30
Sunil Mohan Adapa
2d832ace36
backups: Convert tests to pytest style
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Joseph Nuthalapati <njoseph@thoughtworks.com>
2019-05-12 17:13:22 +05:30
Sunil Mohan Adapa
bbb80fe577
tests: Fix backups API test cases to work under all conditions
Ensure that loaded module list is mocked instead of actually loading modules.

Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
2019-03-10 09:54:09 -04:00
Sunil Mohan Adapa
983f4d13e8
backups: Fix showing not-installed apps in create backup page
- Also minor styling.

Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
2019-01-25 17:59:55 -08:00
Joseph Nuthalapati
32b470bc7c
backups: Automatically install required apps before restore
Fixes #1460

Signed-off-by: Joseph Nuthalapati <njoseph@thoughtworks.com>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
2019-01-25 17:02:05 -08:00
Sunil Mohan Adapa
039461b846
backups: yapf fixes
Reviewed-by: Michael Pimmer <info@fonfon.at>
2018-12-11 23:38:30 +01:00
Sunil Mohan Adapa
f4b0c4dc9f
backups: Don't fail tests when borg is not installed
When a user running tests is root and borg is not installed, several tests fail
due to missing borg. Skip these tests when borg is not installed.

Reviewed-by: Michael Pimmer <info@fonfon.at>
2018-12-11 23:37:38 +01:00
Michael Pimmer
bb95229a91
Backups: unittests for accessing repository with borg directly
- adapt action and write tests for accessing a borg repo directly
  via borg+ssh, without mounting it
- some docstring updates

Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
2018-12-13 18:28:51 -05:00
Michael Pimmer
fc9e406bf8
Backups: allow using keyfile as credentials for sshfs mounts
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
2018-12-13 18:28:44 -05:00
Michael Pimmer
eab8991b54
Backups: support for encrypted repositories
- implement download, restore, delete archives of encrypted
  repositories
- change how BorgRepository and SshBorgRepository handle path
- when/before creating remote repositories, check whether the
  connection works
- updated tests

Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
2018-12-13 18:28:17 -05:00
Michael Pimmer
c223250569
Backups: remove unittests to backups test directory
And add one bugfix to reset cfg.actions_dir after changing it

Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
2018-12-13 18:28:00 -05:00
Michael Pimmer
15e26caa23
Backups, remote repositories: create/delete/restore of remote repos
- updated unittests
- implemented create/delete/restore remote archives

Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
2018-12-13 18:27:48 -05:00
Michael Pimmer
b92b66b7e9
Backups: clean up forms, names and templates
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
2018-11-17 08:59:59 -05:00
Michael Pimmer
c8554bef9d
Backups: fix test
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
2018-11-17 08:59:53 -05:00
Joseph Nuthalapati
c33b192f71
backups: Don't rely on disk labels during export/restore
Disk labels are unreliable. They may not exist. There are not unique and two
disks in the system may have the same label. Instead use the device path of the
disk/partition.

Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
2018-10-22 19:26:22 -04:00
Joseph Nuthalapati
316d765629
backups: Implement app hooks
Each application will be able to implement backup_pre, backup_post, restore_pre
and restore_post hooks that get called before/after backup/restore
appropriately. This is to handle any edge cases that backup manifest mechanism
does not handle.

Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
2018-10-21 19:18:50 -04:00
Joseph Nuthalapati
b18a80f0f2
backups: Implement disabling web configuration during backup
- Introduce a BackupApp class to store all information about application being
  backed up. This cleans up apps lists vs. manifest lists spread out in the
  code.

- Introduce ServiceHandler to abstract dealing with services and web
  configuration.

- Add enable and disable actions in apache action.

Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
2018-10-08 18:28:54 -04:00
Joseph Nuthalapati
8d7ede728e
backups: List apps that don't require backup too
- This serves two purposes. First is to assure user in the interface that backup
  of the module is not required. Second is to make sure that if an application
  is installed during backup it is also reinstalled during restore process (this
  need to be implemented).

- Allow backup test to run independently. Initialize the cfg module so that
  load_modules() works.

Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
2018-10-03 19:18:50 -04:00
Joseph Nuthalapati
26764b7370
backups: Rename test_backup to test_api
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
2018-10-03 19:18:48 -04:00
Michael Pimmer
d8fc2057d4
Backup module: added some unittests; minor doc updates
- added unittests
- introduced function backups.get_location_path()
- minor docstring / comment and naming updates

Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
2018-10-01 07:01:03 -04:00
Joseph Nuthalapati
670f58019e
backups: Minor styling fixes
- More descriptive iteration variables
- Run yafp, isort

Signed-off-by: Joseph Nuthalapati <njoseph@thoughtworks.com>
2018-09-24 19:46:39 +05:30
Joseph Nuthalapati
dc781b03fa
backups: Make getting all apps method public
Signed-off-by: Joseph Nuthalapati <njoseph@thoughtworks.com>
2018-09-24 19:46:18 +05:30
James Valleroy
94a1b37700
backups: Add more basic tests for backups API
Signed-off-by: James Valleroy <jvalleroy@mailbox.org>
Reviewed-by: Joseph Nuthalapati <njoseph@thoughtworks.com>
2018-09-24 19:43:53 +05:30
James Valleroy
3681985f40
backups: Patch actions for shutdown services test
Signed-off-by: James Valleroy <jvalleroy@mailbox.org>
Reviewed-by: Joseph Nuthalapati <njoseph@thoughtworks.com>
2018-09-24 19:41:23 +05:30