- Integrate SSH error handling into borg error handling.
- Move logic to migrate SSH keys into lower levels (Repository class) so that it
can performed at more instances such as when initializing repository. It also
provides better abstraction keeping the view logic simpler.
- Drop ability to mount repository using password. This is important next step
for mounting using systemd unit files.
- Use exceptions to eliminate return value checking.
- Create a special exception for exceptions raised during SSH operations. Raise
this at lower levels and handle these using the common error handler.
Tests:
- Adding a remote repository with key and password authentication works with and
without encryption. Adding works with SSH host key pre-verified works too.
- Trying to add a remote repository with incorrect passpharse fails with the
simplified error message. Redirect happens to add remote repository page. Error
message with SSH host key pre-verified works too. Repository is removed.
- Trying to provide wrong SSH password fails with a simplified error message.
Redirect happens to add remote repository page. Repository is removed.
- Mounting a repository after unmounting it works.
- Mounting a repository with SSH password in its configuration works. Migration
is performed and SSH password is replaced with SSH key file path.
- A schedule for a repository with SSH password runs successfully. An archive is
created. Migration is performed and SSH password is replaced with SSH key file
path.
- SSH identity files are created with plinth:plinth ownership. Private key file
is created with 0o600 permissions and public key file is created with 0o644
permissions.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Tests:
- Adding a remote repository with key and password authentication works with and
without encryption.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Tests:
- On main branch, add a remote repository with SSH password. Unmount
the remote location.
- Switch to branch with this change. Mount the remote location. Logs
show that it is migrated from password to key authentication. Plinth
database no longer contains password for this remote.
Signed-off-by: James Valleroy <jvalleroy@mailbox.org>
- Use javascript to disable or enable password fields.
- If SSH key auth is selected, then try the connection.
- If SSH password auth is selected, then copy the key.
Signed-off-by: James Valleroy <jvalleroy@mailbox.org>
- After copying the SSH client public key to the remote host, replace
the SSH password credential with keyfile.
- Also use SSH key when checking that remote directory exists.
Tests:
- Add remote backup location "tester@localhost:~backups". Test various
operations like create backup, download backup, unmount and
mount. Confirm that SSH password is no longer present in plinth
sqlite database.
Signed-off-by: James Valleroy <jvalleroy@mailbox.org>
Tests:
- In development VM, add a remote backup location of "tester@localhost:~/backups".
Verify the SSH host key. plinth@freedombox key is listed in
/home/tester/.ssh/authorized_keys.
- Remove the remote backup location, and delete /home/tester/.ssh/authorized_keys.
Add the same remote backup location again. plinth@freedombox key is again
listed in /home/tester/.ssh/authorized_keys.
Signed-off-by: James Valleroy <jvalleroy@mailbox.org>
- For scheduled backups and automatic backups generated when uninstalling apps,
make the generated archive name similar to automatic name given no name is
specified when creating an archive.
Tests:
- Set schedule on a local repository, uninstall an application, and create a
backup archive without a name. The timestamp in generated backup archives in all
three cases is similar in format.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Closes: #2354.
- Process spawned for raw IO should be collected using wait(). Closing the
specially created pipe early causes an exception in the child process when it
tries to write to stderr the JSON result of executed method.
Tests:
- Monitor the output of 'journalctl -f'. Download and archive. Without the
patch, an error is printed as described in the bug. With the but no such error
is printed. Downloaded file is the same in both cases.
- Writing a log message in cleanup_func shows that the process has been waited
for and the FDs have been closed.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- When archives are listed using list_archives() method, return datetime objects
instead of pre-formatted strings. datetime objects can be compared easily and
shown in a more human readable format.
Tests:
- Unit tests pass.
- Backups with a future date are ignored when considering recent backup
times (when they logged to console).
- Most recent scheduled backup times are retrieved correctly (when they logged
to console).
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Currently, in the repository class, if an operation fails, the error messages
from borg are interpreted and re-raised. Everywhere else, the errors are
interpreted. Fix this by wrapping privileged methods at the privileged module
level instead of a context manager at the place of calling.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- In current stable and testing, verifying SSH remote hosts using RSA is not
working. After selecting the verified RSA fingerprint, paramiko fails to connect
- A change introduced in paramiko 2.9 lead to failures when connecting to hosts
that have a verified RSA host key[1][2][3]. To fix the issue,
disabled_algorithms must be used to drop some of the other algorithms supported
by the server to force paramiko behavior. A better solution to the problem was
introduced in paramiko 3.2. Both these solutions require careful update to the
code. Considering the utility paramiko provides, the regression annoyance,
effort required for this fix, and the security implications (it is an completely
independent SSH implementation), the library does not seem to be worth the
effort in our case.
- Switch to using sshpass command line utility instead of paramiko library. The
only reason to use paramiko seems that 'ssh' command by default does not allow
us to input password easily while paramiko does.
- Another place where paramiko is being used is to check if a host is already
verified in the known_hosts file. This has been trivially replaced with
'ssh-keygen -F'.
- Exit codes provided by sshpass can replace the specific exception raised by
paramiko.
Links:
1) https://www.paramiko.org/changelog.html
2) https://github.com/paramiko/paramiko/issues/2017
3) https://github.com/paramiko/paramiko/issues/1984
Tests:
- Add a remote backup repository with and without encryption.
- Add remote backup repository with all three types of algorithms.
- Add a remote repository again with wrong password. Authentication error is
properly shown.
- Add a remote backup repository and remove it. Host remains verified. Add a
repository again.
- Add a remote backup repository and remove it. Host remains verified. Change
the fingerprint the /var/lib/plinth/.ssh/known_hosts file. Add a repository
again. A proper error is shown that remote host could not be verified.
- Add a remote backup repository and remove it. Host remains verified. Stop SSH
server on the remote host. A generic error is shown that ssh command on remote
host failed.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Tests:
- With a locale that has translation for the 'FreedomBox' source string, check
the following locations have translated string for 'FreedomBox':
- Default backup repository label.
- Firewall app description.
- Network forms for: topology, internet connection type, and router
configuration.
- RSS-Bridge app description.
- TiddlyWiki app description.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Veiko Aasa <veiko17@disroot.org>
- Try to mark class variables in component classes.
- Leave typing hints generic, such as 'list' and 'dict' where content is usually
not filled, too complex, or context is unimportant.
- backups: Handle failure for tarfile extraction so that methods are not called
on potentially None valued variables.
- backups: Prevent potentially passing a keyword argument twice.
- dynamicdns: Deal properly with outcome of urlparsing.
- ejabberd: Deal with failed regex match
- email: Fix a mypy compliant when iterating a filtered list.
- tor: Don't reuse variables for different typed values.
- tor: Don't reuse variables for different typed values.
- operation: Return None explicitly.
- operation: Ensure that keyword argument is not repeated.
Tests:
- Where only typing hints were modified and no syntax error came up, additional
testing was not done.
- `mypy --ignore-missing-imports .` run successfully.
- Generate developer documentation.
- Service runs without errors upon start up.
- backups: Listing and restoring specific apps from a backup works.
- backups: Mounting a remote backup repository works.
- NOT TESTED: dynamicdns: Migrating from old style configuration works.
- ejabberd: Verify that setting coturn configuration works.
- email: Test that showing configuration from postfix works.
- tor: Orport value is properly shown.
- transmission: Configuration values are properly set.
- users: Running unit tests as root works.
- operation: Operation status messages are show properly during app install.
- ./setup.py install runs
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Tests:
- DONE: Functional tests works
- DONE: Initial setup works
- DONE: Borg repository is created at /var/lib/freedombox/borgbackup
- DONE: With regular and with encrypted repository
- DONE: Creating a repository works
- DONE: Getting information works. When adding a existing location, incorrect
password leads to error in the add form.
- DONE: Listing archives works
- DONE: Creating/restoring an archive works
- DONE: Backup manifest is created in /var/lib/plinth/backups-manifests/
- DONE: Including an app that dumps/restores its settings works
- DONE: Exporting an archive as tar works
- DONE: Exporting a large archive yields reasonable download speeds. 31
MB/s. 1GB file in about 30 seconds.
- DONE: Restoring from an uploaded archive works
- DONE: Listing the apps inside an archive works before restore
- DONE: Errors during operations are re-raises as simpler errors
- DONE: Get info
- DONE: List archives
- DONE: Delete archive (not handled)
- FAIL: Export tar
- DONE: Init repo
- DONE: Get archive apps (not handled)
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Tests:
- Mounting an SSH repository works
- If an known error is thrown during mounting, a simplified error is shown.
- Unmounting an SSH repository works
- If an known error is thrown during mounting, a simplified error is shown.
- Correct status of whether the repository is mounted is shown.
- If an known error is thrown during mounting, a simplified error is shown.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
[sunil: Make the umount code specific to SSH repositories]
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
- ugettext functions will be removed in Django 4.0. Each use emits a warning
when running with Django 3.2. Since we have warnings enabled in developer mode,
we see quite a few messages because of this.
- ugettext is already a simple alias of gettext. So, no regressions are
expected.
Tests:
- Accessing an affected app in UI with Django 3.2 and Django 2.2 works fine.
- Using Django 3.2 there are no warnings related to removal of ugettext
functions.
- Ran regular unit tests.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
This is a simplification of UX by allowing only a single schedule to be attached
to each repository.
Tests performed:
- Editing a schedule works for root, local and remote repositories.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Simplify save() method such that a simple load(uuid).save() does not destroy
data. Currently, the verified parameter is lost.
- Drop unused store_credentials argument to save().
- Make verified a property of the SSH repository and instantiate with
it properly.
Tests performed:
- Adding a new remote repository works with SSH verification from unknown and
known hosts.
- Mounting and unmounting works.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Useful for abstracting the mount() operation that is not available on non-remote
repositories.
- On non-remote repositories do nothing.
- On remote repositories, mount the location in preparation for operations on
the repository.
Tests performed:
- When an SSH repository is unmounted before a schedule backup, it is
automatically mounted for listing, backup and cleanup at scheduled time.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- This allows schedule to be stored along with the repository information.
Tests performed:
- Storing/retrieving a schedule along with root repository works.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Tests performed:
- Schedules are able to store and retrieve comments properly. Information about
schedule backups stored in comments is extracted properly.
- Unit tests run.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Without this error, users will believe that a repository is encrypted (as
password is provided) but it is not (repository was created earlier without a
password).
- Remove the need to send credentials, an instance property, while getting
encryption data, an instance method.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Simplify repository initialization using inheritance.
- Share the code for saving repository for disk and remote repositories. Error
messages are properly handled for disk repositories too.
- Move logic to create remote SSH directory to SSH repository class instead of
views.
- Create a new error for handling borg repository already existing while
initialization.
- Get information of repository after initializing so that password errors are
caught early.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- No need to override run method in derived class if known_credentials is a
class property exposed by base class.
- Move a method for cosmetic.
- Minor formatting.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
The constructor of the Repository object is being used for two distinct
purposes. One is to load the object from database and other to instantiate it
with parameter such that it can be saved to database. Separating the two usages
to different methods simplifies code and parameter passing for consumers.
Also turn some class specific constants from globals to class constants.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
It is being used to mean initialize the borg repository as well as creating an
instance of the repository class object. Use 'initialize' for former and
'get_instance' for latter.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Use repository class instances instead of custom ad-hoc data structures such as
repo_data.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
It is not required by any consumers. Allows for better abstraction on a complex
class.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
borg_path clearly signifies that it is to be used by borg and consumers of the
class will not be confused by it.
Also rename some repo_path variables in test cases.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Since the method is being called on Repository classes, _repository suffix is
redundant.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Make consumers depend on the repository classes instead of lower level store
API.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
get_repositories method will return repositories instead of dictionaries for
view content. This will make it usable in more situations.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Same module is being used to store/retrieve local disk repositories also.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Use sort_order property to decide which type of repositories should be listed
first.
- Remove getting repositories of a given type and retrieve all of them at the
same time.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Also make storage_type an abstract property so that derived classes are forced
to override it.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Remove incorrect instantiation of SSH repository for local disk repositories.
- Clarify the removal message to convey that backups are not removed.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
