- So that on slow machines, the tests don't fail.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Closes: #2218.
- When borg can't acquire a lock due to it being busy, any borg operation can
fail. Show a readable message instead of a generic error.
- Also handle errors for archive already existing and archive to be deleted not
existing.
Tests:
- Functional tests for backup app work.
- Creating archive works with proper message. Providing the name of existing
archive shows proper error.
- Deleting archive works with proper message. Open two tabs by clicking on the
delete archive button. Then delete with one and try to delete the it again with
the next one. Proper error message is shown.
- While downloading an archive, delete an archive. Proper error message that
borg is busy is shown.
- Upload archive works. A proper success message is shown.
- Restore backup from archive works. A proper success message is shown.
- Restore backup from file upload works. A proper success message is shown after
upload and after restoration.
- Adding local repository works. A proper success message is shown.
- Adding remote repository works. A proper success message is shown when SSH key
is verified and repository is added.
- Removing repository works. A proper success message is shown.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Closes: #1603.
- This introduces seconds and timezone in the name of the archive. When seconds
are not used in the name, taking backups too quickly on after the other fails as
the names clash.
Tests:
- Taking a backup works and creates the name in the expected ISO time format.
- Changing the timezone and restarting service creates timestamps in that
timezone.
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- When archives are listed using list_archives() method, return datetime objects
instead of pre-formatted strings. datetime objects can be compared easily and
shown in a more human readable format.
Tests:
- Unit tests pass.
- Backups with a future date are ignored when considering recent backup
times (when they logged to console).
- Most recent scheduled backup times are retrieved correctly (when they logged
to console).
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Tests:
- Sending GET requests to the two views by opening theirs URLs in the browser
show a blank page. Actions are not performed.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Currently, in the repository class, if an operation fails, the error messages
from borg are interpreted and re-raised. Everywhere else, the errors are
interpreted. Fix this by wrapping privileged methods at the privileged module
level instead of a context manager at the place of calling.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Closes: #2476.
- The collapse click logic is triggered when clicking on the schedule button (or
the other buttons). In case of other buttons they do activate eventually but
the schedule button does not activate.
- Solve this properly by making sure that collapse logic is not triggered when
the buttons are clicked.
Tests:
- On stable and testing containers, ensure that collapse works. Collapse/expand
icon works. The three buttons work. And mobile view looks good.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- In current stable and testing, verifying SSH remote hosts using RSA is not
working. After selecting the verified RSA fingerprint, paramiko fails to connect
- A change introduced in paramiko 2.9 lead to failures when connecting to hosts
that have a verified RSA host key[1][2][3]. To fix the issue,
disabled_algorithms must be used to drop some of the other algorithms supported
by the server to force paramiko behavior. A better solution to the problem was
introduced in paramiko 3.2. Both these solutions require careful update to the
code. Considering the utility paramiko provides, the regression annoyance,
effort required for this fix, and the security implications (it is an completely
independent SSH implementation), the library does not seem to be worth the
effort in our case.
- Switch to using sshpass command line utility instead of paramiko library. The
only reason to use paramiko seems that 'ssh' command by default does not allow
us to input password easily while paramiko does.
- Another place where paramiko is being used is to check if a host is already
verified in the known_hosts file. This has been trivially replaced with
'ssh-keygen -F'.
- Exit codes provided by sshpass can replace the specific exception raised by
paramiko.
Links:
1) https://www.paramiko.org/changelog.html
2) https://github.com/paramiko/paramiko/issues/2017
3) https://github.com/paramiko/paramiko/issues/1984
Tests:
- Add a remote backup repository with and without encryption.
- Add remote backup repository with all three types of algorithms.
- Add a remote repository again with wrong password. Authentication error is
properly shown.
- Add a remote backup repository and remove it. Host remains verified. Add a
repository again.
- Add a remote backup repository and remove it. Host remains verified. Change
the fingerprint the /var/lib/plinth/.ssh/known_hosts file. Add a repository
again. A proper error is shown that remote host could not be verified.
- Add a remote backup repository and remove it. Host remains verified. Stop SSH
server on the remote host. A generic error is shown that ssh command on remote
host failed.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- The output of ssh-keyscan scan contain comments that start with '#'. When
these are present, they are incorrectly assumed to be valid keys. Ignore these
lines.
Tests:
- Output of ssh-keyscan with latest OpenSSH in Debian testing contains comments.
Trying to verify the identity of the remove host without the patch fails when
adding remote backup repository. When patch is applied, it works (except RSA
key).
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Ensure that .sr-only is replaced with newer classes.
- Ensure that icons are present for all alerts.
- Use flex-box for display of icons on the left center of the alert.
- .close has been renamed to .btn-close.
- × is no longer required for close buttons.
Tests:
- Visually verify all the changes by triggering them with code changes.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Veiko Aasa <veiko17@disroot.org>
Tests:
- With a locale that has translation for the 'FreedomBox' source string, check
the following locations have translated string for 'FreedomBox':
- Default backup repository label.
- Firewall app description.
- Network forms for: topology, internet connection type, and router
configuration.
- RSS-Bridge app description.
- TiddlyWiki app description.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Veiko Aasa <veiko17@disroot.org>
Fixes: #2364
Tests:
- Set language to English. Go to backups -> create. List of apps is sorted
alphabetically and case is ignored. Take a backup.
- Click on restore for the new backup. The list of apps is again sorted
alphabetically and case is ignored.
- Click on schedules. List of apps is sorted and alphabetically and case is
ignored.
- Repeat tests with Spanish locale.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Veiko Aasa <veiko17@disroot.org>
Fixes: #2115.
Tests:
- Visit the backups upload page. Notice that the new help text is as expected.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Veiko Aasa <veiko17@disroot.org>
Fixes an error in testing container where installing dependencies for
functional tests wants to uninstall system installed urllib3.
Also minor quotes fixes.
Note that dependencies are upgraded only in new dev environments where
geckodriver is not yet installed.
Tests performed, in both stable and testing containers:
- Run all tests, no upgrade related failures found. There were some
seemingly unrelated test failures in apps: calibre, ejabberd, deluged,
minetest, users, transmission , wordpress, zoph.
Signed-off-by: Veiko Aasa <veiko17@disroot.org>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
Tests:
- Run affected privileged actions through UI and notice that secret strings are
not logged (except initializing init/info a new SSH repository).
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Closes: #2161.
- Sections are ordered by importance on which administrator must act after
setting up the system.
- Consistent order across all the languages.
- Update the styling for the section hearers.
- For system section, make them compact.
- Make them look like a header text (with underline) rather than a
divider (like in a menu).
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Veiko Aasa <veiko17@disroot.org>
- This is so that the methods will be checked by mypy. This should help identify
any incorrect initialization of components.
- Remove unused self.repos in GitwebApp.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- It removes this warning. plinth/modules/backups/tests/test_ssh_remotes.py:62: PytestRemovedIn9Warning: Marks applied to fixtures have no effect.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- We stop services before backup and restart them when backup is completed.
However, if backup fails, we are not restarting the services. With this change,
ensure that stopped services are restarted even if backup process fails.
- Similarly for restore operation.
Tests:
- Backup and restore of an app work.
- Functional tests for matrix-synapse work.
- Run the following two tests without the patch to ensure that the reported bug
is reproducible.
- Make a backup operation fail by raising an exception in the privileged code
that takes backup. Enable matrix-synapse app. Run backup including the
matrix-synapse app. Backup fails and shows an error. The service is
stopped before backup and restarted after backup failure.
- Make a restore operation fail by raising an exception in the privileged code
that does restore. Enable matrix-synapse app. Run backup including the
matrix-synapse app and try to restore it. Restore fails and shows an error. The
service is stopped before restore and restarted after restore failure.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Tests:
- Print the interval time in schedule() method and verify that the times are as
expected in develop mode and production mode.
- Notification shows up for RAM usage if the check hardcoded to True.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
borg throws and warning "Warning: Attempting to access a previously unknown
unencryped repository!\nDo you want to continue?" and then fails while asking
for a manual confirmation. Avoid this by telling borg not to prompt (and confirm
yes).
This commit provides a fix for the issue described here:
https://discuss.freedombox.org/t/how-do-i-restore-freedombox/2022/10
[sunil: Drop the part about handling the error message]
Signed-off-by: Benedek Nagy <contact@nbenedek.me>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
Tests:
- mypy does not show any errors.
- Installing ejabberd app works. Privileged actions run fine.
- Unit tests work.
- No additional testing was done as type annotations don't have any effect at
runtime.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Try to mark class variables in component classes.
- Leave typing hints generic, such as 'list' and 'dict' where content is usually
not filled, too complex, or context is unimportant.
- backups: Handle failure for tarfile extraction so that methods are not called
on potentially None valued variables.
- backups: Prevent potentially passing a keyword argument twice.
- dynamicdns: Deal properly with outcome of urlparsing.
- ejabberd: Deal with failed regex match
- email: Fix a mypy compliant when iterating a filtered list.
- tor: Don't reuse variables for different typed values.
- tor: Don't reuse variables for different typed values.
- operation: Return None explicitly.
- operation: Ensure that keyword argument is not repeated.
Tests:
- Where only typing hints were modified and no syntax error came up, additional
testing was not done.
- `mypy --ignore-missing-imports .` run successfully.
- Generate developer documentation.
- Service runs without errors upon start up.
- backups: Listing and restoring specific apps from a backup works.
- backups: Mounting a remote backup repository works.
- NOT TESTED: dynamicdns: Migrating from old style configuration works.
- ejabberd: Verify that setting coturn configuration works.
- email: Test that showing configuration from postfix works.
- tor: Orport value is properly shown.
- transmission: Configuration values are properly set.
- users: Running unit tests as root works.
- operation: Operation status messages are show properly during app install.
- ./setup.py install runs
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- It has been deprecated in Django 4.2
Tests:
- Add a remote backup location with valid SSH hostname and notice that the list
of SSH host keys are shown properly.
- Add a remote backup location with invalid SSH hostname and notice that the
error "Could not reach SSH host..." is shown.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- This will leave /etc/{plinth,freedombox} empty by default making service more
robust to run across various environments and situations. See systemd's
explanation for more details.
- Use Debian maintainer scripts remove all the existing files in
/etc/plinth/modules-enabled.
- Read from /usr/share/freedombox/modules-enabled then from
/etc/plinth/modules-enabled and finally from /etc/freedombox/modules-enabled.
Later read ones override previously read files. Any file pointing to /dev/null
will mean the module must be ignored.
Tests:
- Clean up /etc/plinth, /etc/freedombox and
/usr/share/freedombox/modules-enabled. Run service and notice that files are
getting loaded from development folder using a debug message.
- Run setup.py and notice that files get installed in
/usr/share/freedombox/modules-enabled/ and in the next run they get loaded from
there.
- Create a override file in /etc/plinth/modules-enabled/transmission and notice
that overriden file gets priority over the one in
/usr/share/freedombox/modules-enabled.
- Link the file /etc/plinth/modules-enabled/transmission to /dev/null and notice
that is not loaded.
- Create another file in /etc/freedombox/modules-enabled/transmission and notice
that it overrides the previous two files.
- All affected modules are loaded.
- Build a new Debian package and ensure that upgrading 23.8 to new version
removes are all configuration files.
- Build developer documentation and test that Tutorial -> Full Code and Tutorial
-> Skeleton sections have been updated with references to
-.../modules-enabled/... paths.
- Install quassel and notice that certificates were copied to /var/lib/quassel
directory. Change domain to another domain and notice that certificates were
copied again to that directory.
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Take app_id in a URL fragment and fill that as the default selected app in
create backup form. This URL can be used in apps to create a backup link.
Tests:
- Visit /plinth/sys/backups/create/bepasty/. Only bepasty app will be selected.
- Visit /plinth/sys/backups/create/foo/. No apps are selected.
- Visit /plinth/sys/backups/create/. All apps are selected.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Tests:
- DONE: Functional tests works
- DONE: Initial setup works
- DONE: Borg repository is created at /var/lib/freedombox/borgbackup
- DONE: With regular and with encrypted repository
- DONE: Creating a repository works
- DONE: Getting information works. When adding a existing location, incorrect
password leads to error in the add form.
- DONE: Listing archives works
- DONE: Creating/restoring an archive works
- DONE: Backup manifest is created in /var/lib/plinth/backups-manifests/
- DONE: Including an app that dumps/restores its settings works
- DONE: Exporting an archive as tar works
- DONE: Exporting a large archive yields reasonable download speeds. 31
MB/s. 1GB file in about 30 seconds.
- DONE: Restoring from an uploaded archive works
- DONE: Listing the apps inside an archive works before restore
- DONE: Errors during operations are re-raises as simpler errors
- DONE: Get info
- DONE: List archives
- DONE: Delete archive (not handled)
- FAIL: Export tar
- DONE: Init repo
- DONE: Get archive apps (not handled)
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Tests:
- DONE: Unit tests work
- DONE: Transmission
- DONE: Enabling/disabling an app with a daemon works: transmission
- DONE: Showing the status of whether the app is enabled with daemon
is-enabled works.
- DONE: A message is shown if app is enabled and service is not running
- DONE: Service is stopped and re-started during backup
- DONE: Adding user to share group during initial setup restarts the service
- Not tested: Enabling/disabling a service with alias works (no such apps)
- DONE: Restarting/try-restarting a service works
- DONE: Masking/unmasking works
- DONE: rsyslog is masked after initial setup
- DONE: systemd-journald is try-restarted during initial setup
- DONE: Avahi, email, security initial setup works
- DONE: Fail2ban is unmasked and enabled
- DONE: Enabling/disabling fail2ban is security app works
- DONE: Enabling/disabling password authentication in SSH works
- ?? Let's encrypt
- Services are try-restarted during certificate setup, obtain, renew
- Not tested: upgrade pagekite from version 1
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Tests:
- Mounting an SSH repository works
- If an known error is thrown during mounting, a simplified error is shown.
- Unmounting an SSH repository works
- If an known error is thrown during mounting, a simplified error is shown.
- Correct status of whether the repository is mounted is shown.
- If an known error is thrown during mounting, a simplified error is shown.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Tests:
- Initial setup works when a new container is created
- When transmission is enabled/disabled, the web configuration for it is
enabled/disabled.
- When radicale is enabled/disabled, the uwsgi configuration for it is
enabled/disabled.
- Sharing web configuration is disabled during backup and re-enabled.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Task of managing an operation's progress is now performed by the new Operation
class. Drop them from setup helper.
- Task of providing install() method is now moved to package module. Instead of
storing operation specific data in setup_helper like objects, store them in
thread specific storage that can retrieved anywhere during the operation without
holding references.
- Progress of an operation show as a progress bar is currently missing. This
will be regression until fixed later.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
module.app property usage is greatly reduced because setup() and force_upgrade()
method are now part of App class instead of at the module level. Remove the
remaining minor cases of usage and drop the property altogether.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Primary purpose is to complete the App API and allow for multiple apps to be
present in a module without a single clashing setup() method. Secondary
objective is to get rid of SetupHelper instance simple use App instance instead.
- This brings us closer to not needing to implement setup() method for some of
the typical apps.
- Remove default value None for old_version parameter.
- A valid integer value is always passed to this call.
- The value of None is undefined.
- Simplifies the App API slightly.
- Drop setting 'pre', 'post' values to indicate the stage of setup for the App.
- Simplifies the setup methods significantly. Eliminates a class of
bugs (some of them seen earlier).
- The UI can show a simple 'installing...' or progress spinner instead of
individual stages.
- There are currently many inconsistencies where many operations are not
wrapped in helper.call() calls.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
This is recommended by PEP-0597: https://peps.python.org/pep-0597/
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
[sunil: Make the umount code specific to SSH repositories]
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
- Implemented within the backup component. Scope for implementing database
backup/restore in similar way.
- Add new 'settings' key in the backup manifest to allow keys to backed up and
restored.
- Implement by dumping/loading settings from DB into the file.
Tests:
- Unit tests.
- Backup/restore tests for dynamicdns workss.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
