- When an error occurs during setup thread execution and the error is not due a
failed privileged action, we are left with very little information about what
went run. On the other than when a privileged action fails, we will be logging
the exception twice. But this is okay.
Tests:
- Increment the setup version of one of installed apps and raise an exception in
setup() method. Notice that exception traceback in the logged message.
- Increment the setup version of one of installed apps and raise an exception in
setup's privileged action. Notice that exception traceback in the logged
message twice.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Veiko Aasa <veiko17@disroot.org>
- Operations triggered by FreedomBox service itself such 'apt-get update' and
'apt-get install' don't cause the package operations (post-install and
post-update) to get triggered. This is due to recent implementation of a check
with the FREEDOMBOX_INVOKED environment variable. So, it fairly safe to attempt
these operations immediately as they would have been invoked from outside.
- In one case, when unattended-upgrades is triggered it could lead to
post-install trigger getting triggered too quickly. But this only leads the
operation detecting that apt is busy and performing the long wait immediately
after.
- In case of distribution upgrade, this could mean simpler reasoning and less
wait time.
Tests:
- When a package is installed, post-dpkg operations are triggered and completed
immediately. However, another apt process immediately takes lock, this results
in a waiting period.
- When a 'apt update' is run, update operations are triggered and completed
immediately.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- This allows for the service to become "ready" and serving web connection sooner.
- If some operations such as obtaining certificates and domain configurations
are happening, these can be shown as operations with UI notifications.
Tests:
- Running 'freedombox-develop --setup' works. 'App initialization completed'
message is printed before 'Running setup...' message. Process exits
successfully.
- Running 'freedombox-develop --setup-no-install' works. 'App initialization
completed' message is printed before 'Running setup...' message. Process exits
successfully.
- Running 'freedombox-develop' works. 'App initialization completed' message is
printed before 'Running regular setup' and 'Setup finished'. Cherrypy starts
listening before 'App initialization completed' message.
- Running a fresh VM setup works.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Closes: #1447
Find and rerun setup for apps after a dpkg operation is completed.
This is needed in a couple of situations:
1) Some Debian packages don't manage the database used by the package. When
these packages are updated, their database schema is left at an older version
and service might become unavailable. FreedomBox can perform the database schema
upgrade. However, FreedomBox needs to know when a package has been updated so
that database schema can be upgraded.
2) A package is installed but FreedomBox has not modified its configuration.
Newer version of package becomes available with a new configuration file. Since
the original configuration file has not changed at all, the new configuration
file overwrites the old one and unattended-upgrades deals with this case. Now,
say, the configuration file modifies some defaults that FreedomBox expects
things might break. In this case, FreedomBox can apply the require configuration
changes but it needs to notified as soon as the package has been updated.
When apt runs dpkg, after the operation is completed it triggers commands listed
under the configuration 'Dpkg::Post-Invoke'. This in turn calls this class via a
DBus notification. Here, we iterate through all the apps. If an app is currently
installed and interested in rerunning setup after dpkg operations, then its
setup is rerun. Interest is expressed using the 'rerun_setup_on_upgrade' flag on
the Package() component. If all packages of the app have not be upgraded since
the last check, we skip the operation.
Tests:
- When an app is installed from FreedomBox, the trigger is not run.
- When a package is installed from command line with apt, the trigger is run. It
does nothing.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Closes: #2490
- When app update and force upgrade are pending on an app, app.setup() is run
during initialization. During setup(), force upgrade is first run as expected.
However, force upgrade does not do it's job when an app needs version upgrade.
setup() then tries to run package install() for the app and fails because
configuration file prompt is pending.
Tests:
- On a fresh bookworm container, update all packages. Run freedombox and ensure
that first setup has been completed. Stop freedombox and increment the firewall
app version. Then change sources.list and change bookworm to testing. Run apt
update. Then start the fredombox service. Notice that firewall app setup is run.
During the setup, force upgrader is executed. It install the newer firewall
package with the newer configuration file and performs the configuration file
changes. After that setup process continues and completes successfully.
firewalld package has been upgraded from 1.3.x to 2.3.x. firewalld service is
running. In /etc/firewalld/firewalld.conf default zone is set to external and
backend is set to nftables.
- Rerun the above test without the patches and notice that force upgrader does
not recognize firewall as a package to upgrade and setup() fails when trying to
install() packages. This is run in a loop continuously.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Currently, we are taking a error string and formatting it before it can be
looked up for translation. This causes the lookups to always fail.
- Don't format the error messages and send them as is. Let the
Operation.translate_message and Notification take care of translation.
Formatting will be them after translation. Set the formatting keys as they need
so that exception string is inserted into the message
Tests:
- Set language to Spanish. Through code changes raise an exception in
bepasty.privileged.setup(). Try to install bepasty app. Setup will fail and
error message will shown. The error message will be localized and formatted with
the patch. This is true in the app error message and in the notification.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Not enabled by default currently. This can be changed after further
testing.
- Re-use existing operation from diagnostics run. However, this requires
changing the app_id of the operation for each app.
Tests:
- Enable automatic repair, and run diagnostics. See that repairs are
run.
- Enable automatic repair, and wait for daily diagnostics run. See that
repairs are run.
Closes: #2399.
Signed-off-by: James Valleroy <jvalleroy@mailbox.org>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
- Repair is run within an operation.
- Diagnostics are run for the app first.
- Call app.repair, then re-run setup if needed.
- Add helper functions for apps or components to store error messages in thread
local storage. These error messages are shown at the end.
Signed-off-by: James Valleroy <jvalleroy@mailbox.org>
[sunil: Undo minor reformatting, due to automatic tool]
[sunil: Fix passing incorrect Exception argument to operation.on_update]
[sunil: Add full stop at the end of the success message to match install message]
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
- We have new way to show extra details with exceptions in HTML.
- This handling of error details in PackageException is mostly unused and
unnecessary complexity.
Tests:
- Introduce exception into package.install() and package.uninstall() methods.
Test that exceptions are being shown properly in Django error messages.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
If the package cache is outdated and a check is performed for whether force
upgrade is needed, it might return negative. After the operation proceeds to run
setup, we perform 'apt update' before 'apt install' at this point the
configuration file prompt will cause failure.
Tests:
- Re-run firewalld upgrade from 1.3.x to 2.1.x with a re-run setup.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- This keep the initial setup of the app simple and efficient. Setup will be
less prone to any issues in force upgrade code. There are also fewer chances for
immediate regressions.
Tests:
- Tested as part of the patch series.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- It is okay to instantiate before shutdown to mark shutdown initiation.
- Still keep the instantiating lazy.
Tests:
- Tested as part of patch series.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Avoid running setup if it would bypass a needed force upgrade.
Fixes: #2397
Tests:
- Rerun setup on an app and see that there are no errors.
- Install modified freedombox on bookworm and perform dist-upgrade to
testing. Then rerun setup on Firewall app. It fails with the message "App
firewall failed to force upgrade." firewalld package is not upgraded.
- Modify Firewall app to allow force upgrade to latest version. Then rerun
setup on Firewall app. firewalld is successfully force upgraded.
NOTE: In this case, Firewall setup is run twice, once by force upgrade, and
again by rerun setup.
Signed-off-by: James Valleroy <jvalleroy@mailbox.org>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
- Helps in retrieving an operation that is currently running.
- Prevent starting an operation that is already running.
Tests:
- Unit tests work.
- Installing, uninstalling an app works.
- For upgrading an app works.
- Running background diagnostics works.
- Updating tor configuration works.
- Updating torproxy configuration works.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Tests:
- Install bepasty app. Notice the extra menu option in the advanced menu.
Clicking it installs the app and run setup. Progress is shown during the re-run
of setup. When operation is completed 'App updated' notification is shown.
- Test Zoph app setup page.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Tests:
- DONE: Check if package manager is busy works
- DONE: Power app shows status in app/restart/shutdown pages
- DONE: Upgrades app shows in app page and first boot wizard page
- DONE: When attempting force upgrade, busy state results in a back-off
- DONE: An app's packages can be installed/uninstalled successfully
- DONE: apt update is run before install
- DONE: If network is not available during package install, error message is shown
- DONE: Filtering packages with configuration file prompts works. Tested with
firewall 1.0.3 to 1.2.1.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Currently operations manager already ensures that all operations run serially.
Even in future, operations manager is unlikely to run parallel operations on a
single app.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Allows multiple apps to be queued up for installation. The operation for
installing the package will wait for the package manager to become available.
Wait for 24 hours before giving up.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Task of managing an operation's progress is now performed by the new Operation
class. Drop them from setup helper.
- Task of providing install() method is now moved to package module. Instead of
storing operation specific data in setup_helper like objects, store them in
thread specific storage that can retrieved anywhere during the operation without
holding references.
- Progress of an operation show as a progress bar is currently missing. This
will be regression until fixed later.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- This was required in Python 2 but useless in Python 3.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Make terminology more consistent managed vs. possible, resolve vs. actual.
- Fix regression in security report caused by comparing package expressions with
package names.
- Fix regression in package upgrades caused by comparing package expressions
with package names.
- Update API method names to improve readability and prevent accidental
mismatching of package names and package expressions. Update variable names for
same reason during usage.
Tests:
- minetest install successfully in testing.
- Security report shows non-zero value in the current vulnerabilities column.
- When an unavailable package is added to list of packages in an app, the app
can't be installed.
- When PackageOr expressions is added to an essential package, running
--list-dependencies shows an expressions with '|' in it.
- Unit tests succeed.
- Find a package with conffile prompt and add that to list of a packages in an
app like bepasty and implement a stub force_upgrade() method in the app. Run
'apt update' and that triggers and analysis of packages with conf file prompts.
This should call force_upgrade() method in bepasty and with proper argument for
list of packages.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Warn of installed conflicting packages before installing apps.
[sunil: Rename 'advice' to 'action']
[sunil: Action will be string constant, for better API and i18n]
[sunil: Don't show conflict warning if action is 'ignore']
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
- Also add ability restore missing configuration files during reinstall.
- Reinstall is useful for restoring the original configuration files of the
package.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Don't run the second phase of web framework initialization. This avoids
writing to the DB file.
- Set log level to ERROR so that no messages get printed even to stderr while
listing dependencies.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
Currently, in cases of ignoring an upgrade and actually upgrading, the log
message says success which is somewhat confusing. Make the force_upgrade()
methods in apps return information about ignoring the upgrade and print log
message accordingly.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
This introduces flake8 and fixes a bunch of flake8 errors.
flake8 is run with: ./venv/bin/flake8 plinth
if you're using a python3 venv.
We can eventually further integrate this with gitlab ci.
https://salsa.debian.org/freedombox-team/plinth/issues/58
Reviewed-by: Sunil Mohan Adapa <sunil@medhas.org>
This includes list of packages for which conffile prompts will be shown. For
each package current version of the package, new version of the package and list
of configuration files that were modified.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- So that developers don't have to wait a long time to see the changes.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
- Ensure that force upgrade mechanism runs only once simultaneously.
- Multiple attempts.
- Wait before first attempt and after each attempt. Shutdown properly while
waiting.
- Only consider managed packages of apps that implement force_upgrade() hook.
Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>
Reviewed-by: James Valleroy <jvalleroy@mailbox.org>
